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In his beautiful 2011 book The Most Human 

Human: What Talking to Computers 
Teaches Us About What It Means to Be 
Alive, Brian Christian shares his experi¬ 
ence as a “human confederate” in a quirky 
contest known as the Loebner Prize. It’s 
inspired by Alan Turing’s proposal for how 
to gauge computer intelligence. In each 
round of the Loebner event, human judges 
sit at a computer and carry out text chats 
with two unseen interlocutors. One is a 
human and one is a software program— 
a chatbot. The judges must guess which 
is which, and the chatbot that most often 
gets mistaken for a human wins money for 
its programmer. As an aside, a second title 
is awarded as well, and that’s the one that 
intrigued Christian. It goes to the person 
who was least likely to be mistaken for a 
computer: the most human human. 

You can imagine the task facing a chat¬ 
bot creator: combine language-processing 
code with rules of thumb about how one 
behaves in a conversation, and add just 
enough weirdness to keep the program 
from seeming robotic. But how should a 
human participant demonstrate the fact 
of being human through typed words 
alone? It’s not obvious what aspects of 
your soul you ought to bare, what ineffable 
qualities you could signal that no machine 
could convincingly mimic. 

The ideas Christian wrestled with— 
what cognition is, and what aspects of it 
are exclusively human—are ancient sub¬ 
jects in philosophy. But advances in artifi¬ 
cial intelligence are about to force all of us 
to confront these questions in one way or 
another. What are the technological limits 
constraining how much of our economy 
and our daily lives might be automated? 
What’s the best way to design computers 
so they augment human capabilities, mak¬ 
ing people and machines better together 
than either could be on their own? 

That’s the context for this special issue 
of MIT Technology Review. 

AI is one of the most widely hyped 
technologies, but it’s also easily misun- 
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derstood. To sort things out, we visit a 
pioneer of the much-heralded technique 
known as deep learning (see “Is AI Rid¬ 
ing a One-Trick Pony?” on page 28). We 
explore the effects automation is having 
on labor (“India Warily Eyes AI,” page 38). 
We introduce you to an entrepreneur who 
dreams of using AI to keep people healthy 
(page 46), and we explore how China’s 
investments in the technology could alter 
the global economic order (page 66). 

We offer no doomsday scenarios of 
out-of-control AI or extreme joblessness. 
Those outcomes seem unlikely for rea¬ 
sons Rodney Brooks describes on page 
82. Besides, given the existential threats 
we actually face (environmental catastro¬ 
phe, international conflict), it’s unproduc- 
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tive to put AI on the list as another thing 
to fear. It’s much more likely that people 
will use advanced computing to solve big 
problems, whether it’s by finding medical 
cures, developing greener materials, or 
doing other things we can’t yet envision. 

However, if AI truly is to benefit all of 
humanity and not just exacerbate inequal¬ 
ity, we need to be thoughtful about how it’s 
built and who’s building it, as Fei-Fei Li, 
Tabitha Goldstaub, and Cynthia Dwork 
discuss on pages 26, 45, and 53. And we 
ought to think carefully about how our 
new tools could change us, as Rana el 
Kaliouby (page 8) and Louisa Hall (page 
74) explain. Imagination is required, 
because AI is still in its early days. Very 
human humans can still shape it. 
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For our cover, artist Timothy Luke 
began by rendering an image of a 
stock model’s face in Cinema 4D 
(image 1) and then output a low- 
poly wireframe of the same face 
(2). Using a tool from DeepArt.io, 
based on a convolutional neural 
network, he then ran the render 
against an image of the Mars 
surface obtained from HiRISE, a 
camera on board the Mars Re¬ 
connaissance Orbiter (3), creat¬ 
ing the high-resolution output on 
the right (4). The final image was 
adjusted and converted for print 
(5), then combined into a tryptic 
with the original render image 
elements (6). 
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Views 


COMPUTING 

We Need Computers with 
Empathy 

An emerging trend in artificial intelligence is to get computers to 
detect how we’re feeling and respond accordingly. They might even 
help us develop more compassion for one another. 


I was rehearsing a speech for an Al conference recently when 

I happened to mention Amazon Alexa. At which point Alexa 
woke up and announced: “Playing Selena Gomez.” I had to yell 
“Alexa, stop!” a few times before she even heard me. 

But Alexa was oblivious to my annoyance. Like the majority 
of virtual assistants and other technology out there, she’s clue¬ 
less about what were feeling. 

Were now surrounded by hyper-connected smart devices 
that are autonomous, conversational, and relational, but they’re 
completely devoid of any ability to tell how annoyed or happy 
or depressed we are. And that’s a problem. 

What if, instead, these technologies—smart speakers, 
autonomous vehicles, television sets, connected refrigerators, 
mobile phones—were aware of your emotions? What if they 
sensed nonverbal behavior in real time? Your car might notice 
that you look tired and offer to take the wheel. Your fridge might 
work with you on a healthier diet. Your wearable fitness tracker 
and TV might team up to get you off the couch. Your bathroom 
mirror could sense that you’re stressed and adjust the lighting 
while turning on the right mood-enhancing music. Mood-aware 
technologies would make personalized recommendations and 
encourage people to do things differently, better, or faster. 

Today, an emerging category of AI—artificial emotional 
intelligence, or emotion AI—is focused on developing algo¬ 
rithms that can identify not only basic human emotions such 
as happiness, sadness, and anger but also more complex cog¬ 
nitive states such as fatigue, attention, interest, confusion, dis¬ 
traction, and more. (As for whether machines could not only 
detect emotions but also experience their own, see “How We 
Feel About Robots That Feel,” page 74.) 

My company, Affectiva, is among those working to build 
such systems. We’ve compiled a vast corpus of data consisting 
of six million face videos collected in 87 countries, allowing an 
AI engine to be tuned for real expressions of emotion in the wild 
and to account for cultural differences in emotional expression. 

Using computer vision, speech analysis, and deep learning, 
we classify facial and vocal expressions of emotion. Quite a few 
open challenges remain—how do you train such multi-modal 


Rana el Kaliouby 
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systems? And how do you collect data 
for less frequent emotions, like pride or 
inspiration? 

Nonetheless, the field is progressing 
so fast that I expect the technologies that 
surround us to become emotion-aware 
in the next five years. They will read and 
respond to human cognitive and emo¬ 
tional states, just the way humans do. 
Emotion AI will be ingrained in the tech¬ 
nologies we use every day, running in 
the background, making our tech inter¬ 
actions more personalized, relevant, 


track our mental state, sending alerts to 
a doctor if we chose to share this data. 
Researchers are looking into emotion AI 
for the early diagnosis of disorders such 
as Parkinson’s and coronary artery dis¬ 
ease, as well as suicide prevention and 
autism support. 

Communication: There’s a lot of evidence 
that we already treat our devices, espe¬ 
cially conversational interfaces, the way 
we treat each other. People name their 
social robots, they confide in Siri that 
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I expect the technologies that surround us to become 
emotion-aware in the next five years. 


authentic, and interactive. It’s hard to 
remember now what it was like before 
we had touch interfaces and speech rec¬ 
ognition. Eventually we’ll feel the same 
way about our emotion-aware devices. 

Here are a few of the applications I’m 
most excited about. 

Automotive: An occupant-aware vehicle 
could monitor the driver for fatigue, dis¬ 
traction, and frustration. Beyond safety, 
your car might personalize the in-cab expe¬ 
rience, changing the music or ergonomic 
settings according to who’s in the car. 

Education: In online learning environ¬ 
ments, it is often hard to tell whether 
a student is struggling. By the time test 
scores are lagging, it’s often too late— 
the student has already quit. But what if 
intelligent learning systems could pro¬ 
vide a personalized learning experience? 
These systems would offer a different 
explanation when the student is frus¬ 
trated, slow down in times of confusion, 
or just tell a joke when it’s time to have 
some fun. 

Health care: Just as we can track our 
fitness and physical health, we could 


they were physically abused, and they 
ask a chatbot for moral support as they 
head out for chemotherapy. And that’s 
before we’ve even added empathy. On the 
other hand, we know that younger gen¬ 
erations are losing some ability to empa¬ 
thize because they grow up with digital 
interfaces in which emotion, the main 
dimension of what makes us human, is 
missing. So emotion AI just might bring 
us closer together. 

As with any novel technology, there is 
potential for both good and abuse. It’s 
hard to get more personal than data about 
your emotions. People should have to opt 
in for any kind of data sharing, and they 
should know what the data is being used 
for. We’ll also need to figure out if certain 
applications cross moral lines. We’ll have 
to figure out the rules around privacy and 
ethics. We’ll have to work to avoid build¬ 
ing bias into these applications. But I’m a 
strong believer that the potential for good 
far outweighs the bad. 


Rana el Kaliouby is the CEO and 
cofounder ofAffectiva. In 2012 she was 
named one o/MIT Technology Review’s 
35 Innovators Under 35. 
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Hertz Nazaire is a soft-spoken artist who 

likes to paint in bright colors, with sub¬ 
jects like rainbow palm leaves and danc¬ 
ing women in twirling skirts. But one 
series of paintings he’s created is darker. 
Here, deep-red discs contrast with mis¬ 
shapen, bluish-purple ones against a 
black background. One canvas shows a 
face drowning in the red and blue shapes, 


Sickle-Cell 
Patients See 
Hope in CRISPR 


The disease may be among the first to be 
treated with the novel gene^editing tool. 
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Upfront 


eyes streaming with tears, mouth agape in 
pain. The work reflects his lifelong strug¬ 
gle with sickle-cell disease. 

Nazaire, a 43-year-old Haitian- 
American, figures he’s been hospitalized 
more than 300 times since he was a child. 
He and other sickle-cell patients will tell 
you that the worst part of the disease is 
the debilitating pain. “It’s a horrifying 
thing to have, because it’s extremely pain¬ 
ful. It’s a major fight all the time,” he says. 

Roughly 100,000 people in the U.S. 
have sickle-cell disease, most of them 
African-Americans and Latinos but also 
people of Middle Eastern, Asian, Indian, 
and Mediterranean descent. Compared 
with the average American, they live much 
shorter lives—about 40 to 60 years. 

The cause of sickle-cell has been 
known for a century, but the disease has 
long been underserved by the medical 
establishment and the pharmaceutical 
industry. That may be about to change. 
Its genetic origin—a single, well-studied 
mutation—makes it an attractive candi¬ 
date for treatment with the gene-editing 
tool CRISPR. 

The idea is that CRISPR could cor¬ 
rect the genetic mutation responsible for 
sickle-cell so that patients’ bodies could 
make normal red blood cells, alleviating 
the pain and other severe symptoms asso¬ 
ciated with the disease. Researchers have 
already tested the gene-editing tool on 
human sickle cells in the lab and are now 
working on getting the technique to clini¬ 
cal trials. Early results hint that sickle¬ 
cell could be among the first diseases that 
CRISPR essentially cures. 

Despite the lingering safety concerns 
about using CRISPR in people, some 
sickle-cell patients and their doctors are 
already embracing it. “I would be one of 
the first people to volunteer and say, 1 want 
to take part in a study,”’ Nazaire says. He 
first heard about CRISPR two years ago, 
when he came across a YouTube video fea¬ 


turing Jennifer Doudna and Emmanuelle 
Charpentier, two of the inventors of the 
technology. He’s been enthusiastic about 
the idea of using CRISPR to treat sickle¬ 
cell ever since. 

Sickle-cell disease is one of the most 
common genetic disorders, affecting mil¬ 
lions of people around the world. It’s 
caused by a mutation in a gene known as 
HBB, which makes hemoglobin, a protein 
that transports oxygen throughout the 
body. Blood cells with healthy hemoglobin 
are red and disc-shaped. Cells with abnor¬ 
mal hemoglobin are shaped like sickles 
used to cut wheat, the characteristic that 
gives the disease its name. 

These misshapen cells are sticky and 
clump together. When too many of them 
build up, they create blockages in blood 

“I would be one of the first 
people to volunteer and say, 

‘I want to take part in a study.’” 

vessels and cut off oxygen to nearby parts of 
the body, causing severe episodes of pain. 
The disease can also cause frequent infec¬ 
tions, eye problems, and organ damage. 

CRISPR Therapeutics is one of a 
handful of gene-editing startups pursu¬ 
ing new treatments for sickle-cell. The 
company’s approach involves isolating 
stem cells from samples of patients’ blood. 
Scientists would use CRISPR to activate 
a genetic switch that would raise the lev¬ 
els of a fetal form of hemoglobin in red 
blood cells, turning them healthy. This 
fetal hemoglobin effectively counteracts 
the effects of the sickle mutation. The 
modified cells would then be infused back 
into the patients. 

Samarth Kulkarni, president of 
CRISPR Therapeutics, says this is safer 
than injecting the gene-editing mecha¬ 
nism directly into the patient. That’s risky 


because CRISPR can cause unintentional 
or off-target edits, meaning it may cut 
DNA it isn’t supposed to. Editing cells 
outside the body will allow scientists to 
make sure the technique works before 
reintroducing the cells, he says. 

Testing the method in lab experiments 
using stem cells taken from sickle-cell 
patients, researchers at CRISPR Thera¬ 
peutics found that 85 percent of the cells 
were successfully edited, which means 
they were able to make healthy red blood 
cells. Kulkarni says when the stem cells 
are reintroduced into the patient, they 
should be able to ameliorate all symp¬ 
toms of sickle-cell. These stem cells are 
able to travel to the bone marrow, where 
they make more healthy blood cells for 
the rest of the body. The healthy cells 
will proliferate, and eventually, he says, 
they will outnumber the sickled ones. St. 
Jude Children’s Research Hospital, Editas 
Medicine, and Intellia Therapeutics are 
working on similar approaches. 

Meanwhile, researchers at Stanford 
University School of Medicine are work¬ 
ing on a different method that aims to 
directly modify the mutated HBB gene 
itself using CRISPR. Researchers would 
do that outside the body as well. Matthew 
Porteus, an associate professor of pediat¬ 
rics at Stanford, says his team is aiming to 
begin a clinical trial by early 2019. 

Porteus says not all of a patient’s origi¬ 
nal sickle cells need to be replaced with 
edited ones to effectively cure the disease. 
He says if the proportion of sickle cells 
is below 30 percent, patients don’t have 
any symptoms. So far, his team has been 
able to achieve correction rates between 
40 and 70 percent. He expects corrected 
blood cells to eventually surpass sickled 
ones in a patient’s body. Sickle cells live 
only 10 to 20 days, but normal red blood 
cells last from 90 to 120 days. 

The first clinical trials using CRISPR 
haven’t started in the U.S. yet, but 
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researchers are already taking steps 
to educate patients about the technol¬ 
ogy. The National Institutes of Health is 
launching a study this month to examine 
opinions on CRISPR among up to 150 
sickle-cell patients, parents of patients, 
and health-care providers. 

Vence Bonham, a researcher on genom¬ 
ics and health disparities at the National 
Human Genome Research Institute who is 
leading the study, says it’s important that 
scientists designing clinical trials consider 
patients’beliefs and concerns. Gauging the 
views of people who are most likely to be 
affected by a new scientific advance seems 
like a no-brainer, but it’s something that’s 
rarely been done in medical research. “This 
technology has been moving very quickly, 
but the disease and advocacy communities 
have not really been part of the conversa¬ 
tion,” Bonham says. 

Participants in the NIH study will 
first be asked about their knowledge of 
CRISPR. Then they’ll watch an educa¬ 
tional video about the technology and 
answer a second set of questions to see 
how the video may have influenced their 
knowledge or beliefs. After that, they’ll 
participate in focus groups with other 
patients, parents of patients, or health¬ 
care providers to talk about using CRISPR 
for sickle-cell disease. Bonham hopes the 
study will “inform the development of 
clinical trials to make them more appro¬ 
priate and respectful of the concerns of 
the community.” 

Biree Andemariam, director of the 
New England Sickle Cell Institute at the 
University of Connecticut Health Cen¬ 
ter, has talked to her adult patients about 
CRISPR’s potential for treating sickle-cell 
disease. “Patients are very intrigued by it. 
They think it sounds wonderful,” she says. 

But Andemariam says there can also 
be trust issues between sickle-cell patients 
and their health-care providers. Black 
patients may be suspicious of signing up 


for clinical trials, particularly given his¬ 
torical examples of medical experimenta¬ 
tion on African-Americans without their 
consent. The infamous Tuskegee study, 
for example, left African-American men 
with syphilis deliberately untreated in an 
experiment that ran from 1932 to 1972. 

“The Tuskegee experiment is fresh in 
a lot of people’s minds even though that 
was decades ago,” says Andemariam, who 
is also chief medical officer for the Sickle 
Cell Disease Association of America. 

If a CRISPR cure for sickle-cell even¬ 
tually reaches the market, one major 
question is who will have access to it. Isaac 
Odame, a Ghana-born physician at the 
Hospital for Sick Children in Toronto, 
says patients in Africa already have trou¬ 
ble paying for hydroxyurea, a common 
medication used to treat the disease. The 
drug costs one to two dollars per day, but 
even that is too expensive for many, he 
says. He worries the cost of CRISPR will 


put a cure out of reach for most of the 
world’s patients. 

“For 90 percent of people with sickle¬ 
cell disease who live in this world, this will 
still be something far too expensive for 
them to have access to,” he says. 

Until CRISPR is available, sickle-cell 
patients will have to cope with other treat¬ 
ments. To manage his disease, Nazaire 
recently underwent apheresis, a transfu¬ 
sion procedure that removed and replaced 
some of his red blood cells in an attempt 
to decrease the proportion of sickled ones. 
He’s in less pain than before, but the ben¬ 
efits could wear off over time. 

To Nazaire and others, CRISPR repre¬ 
sents the promise of a better, longer life. 
“When you’re faced with something that’s 
desperate and life-threatening, you want 
to see something done about it,” he says. 
“I think this is something that needs to be 
used. It could be beneficial for the world.” 

—Emily Mullin 
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How Angry Truckers Might 
Sabotage Self-Driving Cars 

Displaced workers armed with “adversarial machine learning” could dazzle 
autonomous vehicles into crashing. 


Before autonomous trucks and taxis hit 

the road, manufacturers will need to solve 
problems far more complex than collision 
avoidance and navigation. These vehicles 
will have to anticipate and defend against 
a full spectrum of malicious attackers 
wielding both traditional cyberattacks 
and a new generation of attacks based on 
so-called adversarial machine learning. 

There have been no reports to date of 
hostile hackers targeting self-driving vehi¬ 
cles. But there were no malicious attack¬ 
ers when the dot-com startups in the 
1990s developed the first e-commerce 
platforms, either. After the first big round 
of e-commerce hacks, Bill Gates wrote a 
memo to Microsoft demanding that the 
company take security seriously. The result: 
today Windows is one of the most secure 
operating systems, and Microsoft spends 


more than a billion dollars annually on 
cybersecurity. Yet hackers keep finding 
problems with Windows operating systems. 

Car companies are likely to go through 
a similar progression. After being widely 
embarrassed by their failure to consider 
security at all, they now appear to be 
paying attention. When hackers demon¬ 
strated that vehicles were vulnerable to 
several specific security threats, automak¬ 
ers responded by recalling and upgrading 
the firmware of millions of cars. Last July, 
GM CEO Mary Barra said that protecting 
cars from a cybersecurity incident “is a 
matter of public safety.” 

But current efforts may be missing the 
next security trend. The computer vision 
and collision avoidance systems under 
development for autonomous vehicles rely 
on complex machine-learning algorithms 


that are not well understood, even by the 
companies that rely on them. 

Last year researchers at Carnegie Mel¬ 
lon University demonstrated that state-of- 
the-art face recognition algorithms could 
be defeated by wearing a pair of clear 
glasses with a funky pattern printed on 
their frames. Something about the pattern 
tipped the algorithm in just the right way, 
and it thought it saw what wasn’t there. 
Also last year, researchers at the Univer¬ 
sity of South Carolina, China’s Zhejiang 
University, and the Chinese security firm 
Qihoo 360 demonstrated that they could 
jam various sensors on a Tesla S, making 
objects invisible to its navigation system. 

Why would anyone want to hack a 
self-driving car, knowing that it could 
cause death? One reason is that autono¬ 
mous vehicles could leave a lot of people 
unemployed, and some of them are going 
to be angry. 

In 2016, Ford CEO Mark Fields said 
that he planned to have fully autonomous 
vehicles operating as urban taxis by 2021. 
Google, Nissan, and others planned to 
have similar autonomous cars on the 
roads as soon as 2020. Those automated 
taxis or delivery vehicles could be vulner¬ 
able to being maliciously dazzled with a 
high-power laser pointer by an out-of- 
work Teamster, a former Uber driver, or 
just a pack of bored teenagers. 

Asked about Uber’s plans for address¬ 
ing the threat of adversarial machine 
learning, Sarah Abboud, a spokesper¬ 
son for the company, responded: “Our 
team of security experts are constantly 
exploring new defenses for the future 
of autonomous vehicles, including data 
integrity and abuse detection. However, 
as autonomous technology evolves, so 
does the threat model, which means 
some of today’s security issues will likely 
differ from those addressed in a truly 
autonomous environment.” 

—Simson Garjinkel 
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Another Way Al Programs Can 
Discriminate Against You 

Voice interfaces, chatbots, and other artificial-intelligence systems can be biased 
against certain minority dialects. 


All too often people make snap judgments 

based on how you speak. Some AI systems 
are also learning to be prejudiced against 
certain dialects. And as language-based 
AI systems become ever more common, 
machines may automatically discrimi¬ 
nate against minorities, warn researchers 
studying the issue. 

Anyone with a strong or unusual 
accent may know what it’s like to have 
trouble being understood by Siri or Alexa. 
This is because voice-recognition systems 
use natural-language technology to parse 
the contents of speech, and it often relies 
on algorithms that have been trained 
with example data. If there aren’t enough 
examples of a particular accent or ver¬ 
nacular, then these systems may simply 
fail to understand you. 

But the problem may be more wide¬ 
spread and pernicious than most people 
realize. Natural-language technology 
now powers interactions with custom¬ 
ers, through automated phone systems or 
chatbots. It’s used to mine public opinion 
on the Web and social networks, and to 


comb through written documents for use¬ 
ful information. This means that services 
and products built on top of language sys¬ 
tems may already be unfairly discriminat¬ 
ing against certain groups. 

Brendan O’Connor, an assistant pro¬ 
fessor at the University of Massachu¬ 
setts, Amherst, and one of his graduate 
students, Su Lin Blodgett, looked at the 
use of language on Twitter. Using demo¬ 
graphic filtering, the researchers collected 
59.2 million tweets with a high probabil¬ 
ity of containing African-American slang 
or vernacular. They then tested several 
natural-language processing tools on this 
data set to see how they would treat the 
statements. They found that one popular 
tool classified these posts as Danish with 
a high level of confidence. 

“If you analyze Twitter for people’s 
opinions on a politician and you’re not 
even considering what African-Americans 
are saying or young adults are saying, that 
seems problematic,” O’Connor says. 

The pair also tested several popular 
machine-learning-based APIs that ana¬ 


lyze text for meaning and sentiment, and 
they found that these systems struggled, 
too. “If you purchase a sentiment analyzer 
from some company, you don’t even know 
what biases it has in it,” O’Connor says. 

He says the problem extends to any 
system that uses language, including 
search engines. 

The issue of unfairness stemming 
from the use of AI algorithms is gaining 
attention as these algorithms are used 
more widely. One example is a propri¬ 
etary algorithm called Compass, which 
is used to decide whether prison inmates 
should be granted parole. The work¬ 
ings of the algorithm are unknown, but 
research suggests it is biased against 
black inmates. 

Some experts say that the problem 
may be affecting a growing number of 
decisions in finance, health care, and edu¬ 
cation. Shared Goel, an assistant profes¬ 
sor at Stanford University who studies 
algorithmic fairness and public policy, 
says the issue is not always straightfor¬ 
ward. He notes that it can be overly sim¬ 
plistic to call algorithms biased: they may 
work entirely as intended, making pre¬ 
dictions that are accurate, and simply 
reflect broader social biases. “It’s better to 
describe what an algorithm is doing, the 
reason it’s doing it, and then to decide if 
that’s what we want it to do,” Goel says. 

—Will Knight 


TO MARKET 


Essential Phone 

Android smartphone 

COMPANY: 

Essential 

PRICE: 

$699 


AVAILABILITY: 

Now 



Many smartphones are odes to metal and glass. This one, from 
Andy Rubin, the creator of Android, has a titanium body and a 
ceramic back (besides looking good, this should hold up to inevi¬ 
table drops). It’s also got two little magnetic connectors on the 
back to snap accessories on and off, so you can add more fea¬ 
tures over time, such as a 360° camera that the company is sell¬ 
ing alongside the phone. Watching videos on the Essential Phone 
is a lot of fun, as its display is fantastic and bright. I was surprised 
that its built-in speaker—which I usually eschew for earbuds— 
worked well even at top volume. But is this phone really “essen¬ 
tial”? For most of us, not so much. —Rachel Metz 
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Percentage of teens 
with up-to-date HPV 
vaccinations 


26-29.9% 

30-39% 

40-49.9% 


50-59.9% 

60% or greater 



► 


The Cancer Vaccine That Too 
Many People Ignore 

been around for more than a 


The potentially life-saving HPV vaccine has 
decade. So why isn’t everyone getting it? 

Mandi Price was a seemingly healthy 

24-year-old college student when, during a 
regular Pap smear, her gynecologist found 
abnormal cells in her cervix. It was stage 2 
cervical cancer. 

Even more devastating was that her 
cancer was preventable. Doctors detected 
a strain of human papillomavirus, the most 
common sexually transmitted infection in 
the U.S., in Price’s cancer cells. That strain 
of HPV is targeted by a vaccine called Gar- 
dasil. But Price never got the vaccine. Her 
primary care doctor in Washington state 
didn’t recommend it when she was a teen¬ 
ager. Had she received it before becoming 


infected with HPV, she might not have got¬ 
ten cancer. 

Price dropped out of school to get treat¬ 
ment. She needed surgery to remove the 
tumor from her cervix; then she underwent 
chemotherapy and radiation to kill any 
remaining cancerous tissue. At her one- 
year follow-up, doctors found that the can¬ 
cer had spread. She endured chemotherapy 
for another six months. Now 29, Price is in 
remission and is working in Los Angeles. 
“Most of my 20s comprised being in a hos¬ 
pital. It was isolating,” she says. 

Merck’s Gardasil vaccine was consid¬ 
ered a breakthrough when it was approved 


by the U.S. Food and Drug Administra¬ 
tion in June 2006. It was the first vac¬ 
cine to protect against several cancers. 

But more than a decade after the vaccine 
came out, rates of use in many places in 
the U.S., especially in the South, Midwest, 
and Appalachian states, still remain much 
lower than rates for other childhood vac¬ 
cines—too low to stop transmission of the 
most dangerous HPV strains. 

In 2016, only about 50 percent of girls 
and 38 percent of boys had all three doses 
of the HPV vaccine needed for full protec¬ 
tion, according to data released recently 
by the U.S. Centers for Disease Control 
and Prevention. HPV causes about 32,000 
cancers every year, with cervical cancer the 
most common for women and oral cancers 
the most frequent in men. 

Electra Paskett, a cancer epidemiolo¬ 
gist at Ohio State University, says she is § 
still surprised that the vaccine’s uptake has g 
been so slow. “It’s crazy that there’s not a 8 
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line around the corner. If we said we have 
a vaccine for breast cancer, we’d be vacci¬ 
nating day and night,” she says. 

The problem the vaccine has faced is 
its link to a taboo in American culture: 
sexual activity among teenagers. About 
one in four people in the U.S., including 
teens, are currently infected with HPV. 
But many health-care providers hesitate 
to recommend the vaccine. 

State vaccination rates were as high as 
73 percent among girls in Rhode Island 
and as low as 31 percent in South Carolina 
for all three doses in 2016. Among boys, 
Wyoming had the lowest rate, with only 
20 percent getting the full round of shots. 
Overall, teens living in major metropolitan 
areas were far more likely to get the vac¬ 
cine than those living in rural areas. 

In some states with low vaccination 
rates, the rates of HPV-caused cancers 
are the among the highest. In Mississippi, 
only about 34 percent of girls and 25 per¬ 
cent of boys get all required doses of the 
vaccine. The state also has one of the high¬ 
est rates of HPV-related cervical cancer in 
the country. Wyoming tells a similar story, 
with high rates of HPV-associated cancers 
in both men and women. 

HPV vaccination for boys is especially 
lagging in some areas. Paskett, who has 
studied cancer in Appalachia, say there’s 
a perception that HPV only causes can¬ 
cers in women. “A lot 
of parents don’t know 
that boys should be vac¬ 
cinated,” she says. 

A 2015 study found 
that a quarter of the 776 
pediatricians and family physicians sur¬ 
veyed do not strongly endorse the HPV 
vaccine. A third of the doctors said that 
having to talk about a sexually transmit¬ 
ted infection makes them uncomfortable. 

“We don’t usually explain to patients 
how they get hepatitis or meningitis,” says 
Nneka Holder, associate professor of ado¬ 


lescent medicine at the University of Mis¬ 
sissippi Medical Center. “So why should 
HPV be different?” Instead, she says, 
health-care providers should focus on the 
cancer prevention aspect of the vaccine, 
rather than on how HPV is spread. 

Since the vaccine is just over 10 years 
old, it’s too early to know how many cases 
of cancer it has prevented. But clinical tri¬ 
als have shown that the vaccine provides 
nearly 100 percent protection against cer¬ 
vical infections caused by some strains of 
HPV. These infections have fallen by 64 
percent among teen girls in the U.S. since 
the vaccine was introduced. Clinical trials 
of the HPV vaccine have also shown it’s 
safe for both boys and girls. 

These benefits have led Virginia, Rhode 
Island, and Washington, D.C., to adopt 
public school mandates for HPV vaccina¬ 
tion. But some parents are still uncomfort¬ 
able about the HPV vaccine’s association 
with sex and think their children don’t 
need it because they’re not sexually active. 

Aimee Gardiner, director of a group 
called Rhode Island Against Mandated 
HPV Vaccine, says she doesn’t see HPV 
as the “epidemic” she thinks the CDC has 
made it out to be. “For me, the risk of devel¬ 
oping a cancer from any HPV is so insig¬ 
nificantly small that I do not feel like the 
vaccine is a necessity,” she says. 

It’s true that for most people, the 
immune system clears 
the virus from their 
bodies naturally. But 
for a small number of 
people, HPV persists 
and can turn cancer¬ 
ous. For those patients, like Price, the 
result can be a major ordeal, not to men¬ 
tion much more expensive than a vaccine 
that costs about $150 per dose. 

“I am a huge proponent of it,” she says. 
“If you had the chance to prevent cancer 
in your son or daughter, why wouldn’t you 
do that?” —Emily Mullin 


“We don’t usually explain 
to patients how they get 
hepatitis or meningitis.” 


QUOTED 


“You can’t have a data set 
that includes the whole 
world.” 

—Yibiao Zhao, cofounder of iSee, on the 
importance of giving self-driving cars a kind of 
common sense and the ability to quickly deal with 
new situations. 


“It generates really 
interesting-looking cats.” 

—Douglas Eck, a research scientist at Google, on 
Sketch-RNN, a neural network that can generate 
its own drawings. 


“There is no basis for 
saying our findings are 
impossible.” 

—Saswati Chatterjee, scientific founder 
of Homology Medicines, on critics of the 
company’s claim that it can repair human genes 
with viruses, a potential improvement over the 
gene-editing tool CRISPR. 


BY THE NUMBERS 


1,101 miles 

Distance that a Proterra electric bus traveled on a 
single charge—a world record, albeit at 15 mph. 


12 inches 

Distance over which a startup called Pi claims 
its wireless charger can send power to a mobile 
device by inductive charging. 


1 trillion 

The number of connected devices that ARM 
thinks its chips could enable by 2035. 


52 percent 

The proportion of people on the planet who still 
lack Internet access, according to a UN report. 


20 

According to Symantec, the number of cases 
across the U.S., Turkey, and Switzerland in which 
criminals gained operational access to energy 
facilities—in theory, giving them the ability to 
remotely turn off devices like circuit breakers. 
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Potential Carbon Capture 
Game Changer Is Near 

If it works as expected, the Net Power natural-gas demonstration plant will 
capture carbon at nearly no cost. 


On a small lot between Houston and the 

Gulf Coast, in an industrial zone packed 
with petrochemical factories and gas pipe¬ 
lines, a little-known company is finalizing 
construction of a demonstration power 
plant that could represent a genuine energy 
breakthrough. If it works as expected, 
Net Power’s $140 million, 50-megawatt 
natural-gas plant will capture effectively all 
of the carbon dioxide it produces, without 
significantly higher costs, in part by rely¬ 


ing on the greenhouse gas itself to crank 
the turbine that generates electricity. The 
technology could enable a new generation 
of plants that provide clean power, with¬ 
out the development risks of nuclear, the 
geographic restrictions of hydroelectric, or 
the intermittency issues of solar and wind. 

Net Power’s maze of tubes, tanks, com¬ 
pressors, and pumps uses what is known 
as the Allam Cycle. It eliminates the steam 
cycle by replacing water with supercriti¬ 


cal carbon dioxide. In this state, achieved 
under high heat and pressure, carbon diox¬ 
ide takes on the properties of both a liquid 
and a gas. The process was primarily devel¬ 
oped by the British chemical engineer and 
inventor Rodney Allam, now a partner and 
chief technologist at the technology invest¬ 
ment and development firm 8 Rivers Capi¬ 
tal in Durham, North Carolina. Net Power 
is a collaboration between 8 Rivers, plant 
operator Exelon Generation, and energy 
construction firm CB&I. 

Eventually, Net Power expects to 
produce electricity for around $42 per 
megawatt-hour, on par with combined- 
cycle natural gas without carbon capture. 
But on top of all that, the company can 
also sell several by-products, including 
carbon dioxide. Taken together, those 
could effectively push energy production 
costs down to around $20 per megawatt- 
hour (theoretically, even as low as $9 per 
megawatt-hour). Local, state, or federal 
clean-energy policies—such as carbon 
taxes, cap-and-trade systems, and emis¬ 
sions standards—could further improve 
the economics. 

Net Power has already begun conver¬ 
sations and explored potential sites for its 
first commercial plant, which it hopes will 
come online as early as 2021. “If it plays out 
as advertised, it could be an actual game 
changer,” says Jesse Jenkins, a researcher at 
the MIT Energy Initiative. —James Temple 


TO MARKET 


Kymriah 

Gene therapy for cancer 

COMPANY: 

Novartis 


PRICE: 

$475,000 

AVAILABILITY: 

Now 



A revolutionary cancer therapy that uses genetically engineered 
immune cells was approved in August by the U.S. Food and Drug 
Administration. Kymriah, which the FDA calls the “first gene 
therapy” in the U.S., is designed to treat an often lethal type of 
childhood blood and bone marrow cancer. “We’re entering a 
new frontier in medical innovation with the ability to reprogram 
a patient’s own cells to attack a deadly cancer,” said FDA com¬ 
missioner Scott Gottlieb in a statement. It costs $475,000, which 
David Mitchell, founder of the advocacy group Patients for Afford¬ 
able Drugs, calls “excessive.” Novartis says there will be no charge 
if a patient does not respond within one month. —Emily Mullin 
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China’s Ambitious Push for 
Geoengineering Research 

Backed by $3 million in federal funds, Chinese scientists are assessing how 
geoengineering would affect agriculture, glaciers, sea levels, and more. 


During the last three years, China has 

assembled one of the largest federally 
funded geoengineering research programs 
in the world, marking another area where 
it’s forging ahead of other nations on cli¬ 
mate matters. 

The approximately $3 million pro¬ 
gram, funded by the Ministry of Science 
and Technology, incorporates around 15 
faculty members and 40 students across 
three institutions. The researchers are 
assessing the impact of using technology 
to alter the climate and exploring related 
governance issues. The effort explicitly 
does not include technology development 
or outdoor experiments, in contrast to 
emerging U.S. research programs at Har¬ 
vard and the University of Washington. 

“They don’t want to be seen as the bad 
guys, so there’s a reluctance to do that 


among some groups,” says John Moore, a 
British glaciologist and climate modeler 
who is overseeing the program. 

Geoengineering is a blanket term for 
a number of proposed methods for coun¬ 
teracting climate change. Among other 
approaches, scientists have explored the 
possibility of spraying particles into the 
stratosphere to scatter sunlight. Another 
idea is making coastal clouds more reflec¬ 
tive. It’s generally believed such methods 
could offset temperature increases, but 
there are considerable concerns about 
potential environmental side effects, the 
tricky political challenges they raise, and 
the ethics of deploying a technology that 
could alter climate on a global scale. 

Given these challenges and the ris¬ 
ing threat of climate change, a growing 
number of scientists say there should be 


far more research and debate on all these 
issues. And since geoengineering would 
generally affect all countries, regardless 
of which ones deploy it, the more nations 
that take part the better, says Douglas 
MacMartin, a senior research associate 
in mechanical and aerospace engineering 
at Cornell University, who has advised on 
the Chinese program. 

Because China is increasingly influ¬ 
ential on climate issues, the broader sig¬ 
nificance of the geoengineering program 
may be the international example that it 
sets, says Janos Pasztor, executive director 
of the Carnegie Climate Geoengineering 
Governance Initiative. 

Notably, it could compel other 
nations to make similar investments in 
exploring the regional impacts and pol¬ 
icy implications of geoengineering, or at 
least to engage on the subject. In fact, the 
program has already taken steps to incor¬ 
porate poor nations like the Philippines 
and Bangladesh into discussions on the 
issue; this past summer it hosted a work¬ 
shop on geoengineering in the develop¬ 
ing world. 

But are there any risks if China, with 
its mixed history on human rights and 
its lack of democratic institutions, estab¬ 
lishes itself as a scientific leader in a field 
with the power to alter the entire globe, 
for better or worse? 

To the degree that the science is openly 
published and there’s a lot left to learn, “it 
doesn’t matter which country understands 
first,” Pasztor says. 

But over the long term, there could be 
risks if any single nation dominates the 
research on this issue, because it could 
also come to dominate the debate on how, 
when, and whether to ultimately deploy 
such technology, MacMartin says. 

“It is essential for the U.S. to continue 
to have a seat at the table in any decision 
that affects the entire planet,” he says. 

—James Temple 
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Apple Wireless Earbuds Could 
Double as Hearing Aids 

This app that pairs with AirPods promises to help people with hearing loss. 


What if a hearing aid could be replaced with 

a pair of wireless earbuds and a smart¬ 
phone app? 

A Swiss startup is trying to make this a 
reality with an app called Fennex, recently 
released for the iPhone, that works with 
Apple’s $159 AirPods wireless earbuds. 
Alex Mari, CEO of the startup of the 
same name, says that he chose Apple’s 
devices and mobile platform for the app 
in part because of their popularity but 
also because he thinks an Android phone 
would result in more latency when pro¬ 
cessing sound. 

The app is still in its earliest days; it’s 
currently free, though Fennex may even¬ 
tually charge for certain features. Mari 
says today’s version functions like a “cheap 
hearing aid”: it simply tests your hearing 
in each ear and uses those results to act 
as a personalized, adjustable amplifier. If 


you’re having trouble hearing in a class, for 
instance, you could place your phone near 
the lectern while you’re sitting a few rows 
back and listening in on a pair of AirPods. 

But upgrades are coming, Mari says: 
the app is slated to gain features that will 
help reduce unwanted noise and feed¬ 
back. And beyond helping people who 
just want to hear better in some situa¬ 
tions, the software could eventually work 
with Apple’s hardware to serve as a viable 
alternative to a regular hearing aid for 
people who have moderate hearing loss, 
he believes. 

“We want to get as close as we can to 
hearing aid technology,” he says. 

Fennex is latching onto a broader 
trend in using smartphones and earbuds 
to augment hearing, both for people with 
hearing loss and for those who simply 
want to adjust how they hear. Already, 


there are similar apps that work with ear¬ 
buds and smartphones, like Petralex, as 
well as specialized earbuds that work with 
their own smartphone apps, like Dop¬ 
pler Labs’ Here One. Unlike hearing aids, 
which can cost thousands of dollars for a 
pair, these so-called hearables may cost 
up to a few hundred dollars on top of the 
price of a smartphone. 

Fennex claims to have the only app 
designed for AirPods specifically. And 
since Apple is clearly moving away from 
the traditional headphone jack—it cut it 
out of the iPhones released last year and 
included just its proprietary Lightning 
connector for charging—it is possible that 
in the coming years AirPods will become 
a lot more common. 

Larry Humes, a distinguished pro¬ 
fessor of speech and hearing sciences at 
Indiana University Bloomington, is opti¬ 
mistic that something like Fennex could 
take the place of a traditional hearing aid 
for those with mild to moderate hearing 
loss. He thinks the recent passage of leg¬ 
islation that orders the FDA to create a 
new class of over-the-counter hearing aids 
could help, too, as he expects it will lead 
to a range of hearing-related products 
that people can get without needing to 
visit a doctor. 

However, as Mari knows, it’s clearly a 
risk to focus on making software for one 
pair of earbuds, since Fennex can’t control 
what Apple does with its hardware or the 
access it gives to developers who want to 
develop software that works with it. 

There’s also the issue of hear¬ 
ing delays. Fennex works by using the 
AirPods’ microphones to record sounds in 
the world around the wearer, after which 
it sends the audio to the iPhone app for 
processing and then back to the earbuds. 
The delay, for now, is 130 milliseconds, 
which might be fine for listening but is 
noticeable if you’re trying to engage in a 
conversation. —Rachel Metz 
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The App That Boomed in China 
(but Bombed in the U.S.) 

China’s WeChat struggles for an audience overseas. 


Some of America’s most formidable tech¬ 
nology companies have gone to China and 
found that their superpowers seemed to 
vanish. Uber, Amazon, Apple—all were 
humbled by the Chinese market. It may 
be tempting to blame Chinese government 
restrictions, but that wouldn’t explain why 
the reverse is also true: China’s superstar 
companies struggle to make it in the U.S. 

There’s no better example than Ten¬ 
cent’s WeChat, a mobile messaging and 
payment app with over 900 million 
monthly active users. Foreigners who 
visit China are awestruck by the complete 
dominance of this app, which people use 
for everything from sending messages to 
hailing taxis to buying real estate. 

In 2013, Tencent announced that it 
aimed to bring WeChat to America. There 
were already messaging apps in the U.S. 
market, but none had taken the country 
by storm, and there seemed to be room for 
WeChat to shake things up. 

It didn’t happen. But why? It’s no 
secret that China has a heavily censored 
Internet and that WeChat is a heavily 
censored product. This can make inter¬ 



national expansion confusing at best: sen¬ 
sitive content on international accounts 
might be blocked in China. But censorship 
doesn’t appear to be the main obstacle for 
WeChat in the U.S. Rather, WeChat faces 
the same network-effect challenges that 
American companies encounter in China. 
Facebook is currently blocked there, a fac¬ 
tor that certainly contributed to WeChat’s 
success. Now, if Facebook were to make it 
in, it would find that everyone is already 
on WeChat. The two products are dif¬ 
ferent, of course, but WeChat works so 
well in China precisely because so many 


Chinese people are on it. Users’ favorite 
brands and celebrities are there, as are 
friends, family, and coworkers. Facebook 
would have a nearly impossible time rep¬ 
licating that kind of network. 

Network effects are only part of the 
story. WeChat was designed for the Chi¬ 
nese market. Connie Chan, a partner at 
Andreessen Horowitz, says that when 
WeChat was launched in 2011, Chinese 
people had no universal way to commu¬ 
nicate. E-mail penetration was low. SMS 
and voice plans were relatively expensive, 
and SMS spam was a serious problem. By 
the end of 2015, more than half of China’s 
population was online, with 90 percent 
accessing the Web by smartphone. 

This was very different from the envi¬ 
ronment that Tencent encountered in the 
U.S. in 2013. People already had Face- 
book, WhatsApp, and iMessage, as well 
as e-mail and many other ways to com¬ 
municate. Chan explains, “There was less 
of a strong need to have this one unifying 
platform for everyone.” 

Companies like Tencent “want to be 
compared to Facebook, Google, and Ama¬ 
zon,” Chan says. “They want to be in that 
same conversation.” She notes that the 
market cap for Tencent is over $390 bil¬ 
lion: “From that perspective, they have 
the capital to go overseas and purchase 
things, or invest in things, or try more 
experiments.” —Emily Parker 
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You’re a construction worker and you’re exposed to carbon 
monoxide, or you fall into a hole when nobody’s around. Nor¬ 
mally that means you’re in deep trouble. But what if sensors on 
your hard hat could monitor the air quality in real time, or monitor 
your location and alert people nearby? That’s the idea behind the 
GuardHat, a sensor-laden hard hat from a startup of the same 
name. Gerrit Reepmeyer, the company’s cofounder, says the 
system is currently being tested by three oil and gas companies 
in North America and Europe, with more tests by mining opera¬ 
tions on the way. The flip side for workers: wherever you go, your 
supervisor knows about it. —Timothy Maher 
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The System Behind Bitcoin Is 
Easing the Plight of Refugees 

Finland’s digital money system for asylum seekers shows what blockchain 
technology can offer the unbanked. 


For a refugee in a new country, identity, at 

least in the official sense, can be among 
the hardest things to recover. And with¬ 
out an official ID it is nearly impossible 
to advance in society. 

Finland, which like many European 
nations has recently seen a large influx of 
asylum seekers, is using a cryptographic 
ledger, or blockchain, to help them get on 
their feet faster. 

For two years the Finnish Immigra¬ 
tion Service has been giving asylum seek¬ 
ers who don’t have bank accounts prepaid 
MasterCards instead of the traditional 
cash disbursements, and today the pro¬ 
gram has several thousand active card¬ 
holders. Developed by the Helsinki-based 
startup Moni, the card is also linked to a 
unique digital identity stored on a block- 
chain, the same technology that underpins 
the digital currency Bitcoin. 

Bitcoin has demonstrated how block- 
chain technology can be used to transmit 
value between people, with no need for 
corporate middlemen. Central to the tech¬ 
nology is a software protocol that creates a 
permanent record of every Bitcoin transac¬ 
tion. Anyone can access this record, called 
the blockchain, by downloading the Bitcoin 
software. Computers running the software 
all over the world maintain the blockchain, 
and use it to verify new transactions. 

Blockchains are a promising avenue 
for opening new financial opportunities 
to people who don’t have access to modern 
financial services. Besides eliminating the 
need for a traditional financial institution 
to mediate transactions, they provide a way 
to create and securely store a digital form of 


identification that can’t be corrupted and is 
easily accessible from anywhere. 

In Finland, a Moni card can help 
address several challenges facing asylum 
seekers, says Jouko Salonen, director of 
the Finnish Immigration Service. A Moni 
account functions like a bank account. Peo¬ 
ple can use their accounts to buy things, 
pay bills, and even receive direct deposits 
from employers. Meanwhile, every trans¬ 
action is recorded in a public, virtually 
incorruptible database maintained by a 
decentralized global network of computers. 

The technology helps unbanked asy¬ 
lum seekers because what typically keeps 
them from getting bank accounts and jobs 
is that they’re missing a form of strongly 
authenticated identity, says Salonen. He 
adds, “We have found a way to solve that.” 

Moni’s technology uses one of a num¬ 
ber of public blockchains as the means of 
transferring value—but in a way that to the 
users seems like using a debit card. A card¬ 
holder can pay for things at MasterCard 
terminals, or enter a number into a Web 
form to make payments online. 

Antti Pennanen, Moni’s founder and 
CEO, likens blockchain to the early Inter¬ 
net, which was accessible only to a select 
few with the technical wherewithal to use 
it. His says Moni’s technology is somewhat 
analogous to a modem, which made the 
Internet usable for more people. 

Pennanen says word of Moni has 
spread to refugee camps throughout 
Europe. He expects substantial demand 
among those displaced communities once 
the service is available in other countries. 

—Mike Orcutt 
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Exciting, but Not Very Deep 



• r**f<*Q *<*n • tvrrurs muvi ooka mhmi ♦ • 

TechnologyReview 


A debate centers on whether expert 

systems enable computers to really 
think. Nobel laureate Herbert Simon, 
an Al pioneer, believes they do. “Intel¬ 
ligence has to do with the ability to 
respond appropriately to complex sit¬ 
uations,” he says. “Every time we write 
a computer program to do that, par¬ 
ticularly if it’s a fairly general program, 

I find it quite natural to speak of the 
computer as exhibiting intelligence.” 
Marvin Minsky of MIT thinks the pres¬ 
ent expert systems have gone as far 
as they can go, and Al researchers are 
going to have to go back to basics, 
or “First Principles.” “These programs 
are exciting but most of them are not 
very deep,” he says. “You don’t see 
researchers working on the prob¬ 
lem of commonsense reasoning, for 
instance. There is no program around 
today that will tell the difference 
between a dish and a cup.” 

Excerpted from ‘‘The Practical Face of 
Artificial Intelligence,” by Stanford pro¬ 
fessor Joel N. Shurkin in the November 
1983 issue of Technology Review. 
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Q+A 


Fei-Fei Li 


As the director of Stanford’s Al 
Lab and now as a chief scientist 
of Google Cloud, Fei-Fei Li is 
helping to spur the Al revolution. 
But it’s a revolution that needs to 
include more people. She spoke 
with MIT Technology Review 
senior editor Will Knight about 
why everyone benefits if we 
emphasize the human side of 
the technology. 


Why did you join Google? 

Researching cutting-edge Al is very satisfying and rewarding, but were seeing this 
great awakening, a great moment in history. For me it’s very important to think about 
AI’s impact in the world, and one of the most important missions is to democratize 
this technology. The cloud is this gigantic computing vehicle that delivers computing 
services to every single industry. 

What have you learned so far? 

We need to be much more human-centered. If you look at where we are in Al, I would 
say it’s the great triumph of pattern recognition. It is very task-focused, it lacks con¬ 
textual awareness, and it lacks the kind of flexible learning that humans have. We also 
want to make technology that makes humans’ lives better, our world safer, our lives 
more productive and better. All this requires a layer of human-level communication 
and collaboration. 

How can we make Al more human-centered? 

There’s a great phrase, written in the ’70s: “the definition of today’s Al is a machine 
that can make a perfect chess move while the room is on fire.” It really speaks to the 
limitations of AL In the next wave of Al research, if we want to make more helpful and 
useful machines, we’ve got to bring back the contextual understanding. We’ve got to 
bring knowledge abstraction and reasoning. These are all the most important steps. 


At Stanford you created Visual Genome, a database of images that are extensively 
labeled so they can be used for Al systems. Is this interplay of vision and language 
necessary for the next leap forward? 

Absolutely. Vision is a cornerstone of intelligence, and language understanding is a 
cornerstone of intelligence. What makes humans unique is that evolution gave us the 
most incredible and sophisticated vision system, motor system, and language system, 
and they all work together. Visual Genome is exactly the kind of project that’s pushing 
the boundaries of language understanding and visual understanding. And eventually 
we’re going to connect with the world of robotics as well. 


You’ve talked about the need to have more women involved in Al. Why? 

More jobs will be related to artificial intelligence, so we need a huge workforce, and we 
need a more inclusive base. That’s an economic argument. There are also tons of stud¬ 
ies that have shown that when a diverse group of workers come together, the solutions 
they find in their work are more innovative and more creative. That drives innovation. 
But it’s also moral and ethical. 

When you are making a technology this pervasive and this important for humanity, 
you want it to carry the values of the entire humanity, and serve the needs of the entire 
humanity. If the developers of this technology do not represent all walks of life, it is very 
likely that this will be a biased technology. I say this as a technologist, a researcher, and 
a mother. And we need to be speaking about this clearly and loudly. 
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IsAI 
Riding a 
One-Trick 
Pony? 

Self-driving cars, computers that win 
Go championships, and just about 
every other Al advance you’ve heard of 
all depend on a breakthrough that’s 
three decades old. Keeping up the 
pace of progress will require 
confronting Al’s serious limitations. 

By James Somers 


This publication from the mid-1980s showed 
how to train a neural network with many layers. 
It set the stage for this decade’s progress in Al. 


I’m standing in what is soon to be the center of the world, 

or is perhaps just a very large room on the seventh floor of a 
gleaming tower in downtown Toronto. Showing me around is 
Jordan Jacobs, who cofounded this place: the nascent Vector 
Institute, which opens its doors this fall and which is aiming to 
become the global epicenter of artificial intelligence. 

Were in Toronto because Geoffrey Hinton is in Toronto, and 
Hinton is the father of “deep learning,” the technique behind the 
current excitement about AL “In 30 years were going to look 
back and say Geoff is Einstein—of Al, deep learning, the thing 
that we’re calling Al,” Jacobs says. Of the Al researchers at the 
top of the field of deep learning, Hinton has more citations than 
the next three combined. His students and postdocs have gone 
on to run the Al labs at Apple, Facebook, and OpenAI; Hinton 
himself is a lead scientist on the Google Brain Al team. In fact, 
nearly every achievement in the last decade of Al—in transla¬ 
tion, speech recognition, image recognition, and game playing- 
traces in some way back to Hinton’s work. 

The Vector Institute, this monument to the ascent of 
Hinton’s ideas, is a research center where companies from 
around the U.S. and Canada—like Google, and Uber, and 
Nvidia—will sponsor efforts to commercialize Al technologies. 
Money has poured in faster than Jacobs could ask for it; two 
of his cofounders surveyed companies in the Toronto area, and 
the demand for Al experts ended up being 10 times what Can¬ 
ada produces every year. Vector is in a sense ground zero for 
the now-worldwide attempt to mobilize around deep learning: 
to cash in on the technique, to teach it, to refine and apply it. 
Data centers are being built, towers are being filled with start¬ 
ups, a whole generation of students is going into the field. 

The impression you get standing on the Vector floor, bare 
and echoey and about to be filled, is that you’re at the begin¬ 
ning of something. But the peculiar thing about deep learn¬ 
ing is just how old its key ideas are. Hinton’s breakthrough 
paper, with colleagues David Rumelhart and Ronald Williams, 
was published in 1986. The paper elaborated on a technique 
called backpropagation, or backprop for short. Backprop, in the 
words of Jon Cohen, a computational psychologist at Princeton, 
is “what all of deep learning is based on—literally everything.” 

When you boil it down, Al today is deep learning, and deep 
learning is backprop—which is amazing, considering that 
backprop is more than 30 years old. It’s worth understand¬ 
ing how that happened—how a technique could lie in wait 
for so long and then cause such an explosion—because once 
you understand the story of backprop, you’ll start to under¬ 
stand the current moment in Al, and in particular the fact 
that maybe we’re not actually at the beginning of a revolution. 
Maybe we’re at the end of one. 
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Vindication 

The walk from the Vector Institute to Hinton’s office at Google, 
where he spends most of his time (he is now an emeritus pro¬ 
fessor at the University of Toronto), is a kind of living adver¬ 
tisement for the city, at least in the summertime. You can 
understand why Hinton, who is originally from the U.K., 
moved here in the 1980s after working at Carnegie Mellon 
University in Pittsburgh. 

When you step outside, even downtown near the financial 
district, you feel as though you’ve actually gone into nature. It’s 
the smell, I think: wet loam in the air. Toronto was built on top 
of forested ravines, and it’s said to be “a city within a park”; as 
it’s been urbanized, the local government has set strict restric¬ 
tions to maintain the tree canopy. As you’re flying in, the outer 
parts of the city look almost cartoonishly lush. 

Toronto is the fourth-largest city in North America (after 
Mexico City, New York, and L.A.), and its most diverse: more 
than half the population was born outside Canada. You can 
see that walking around. The crowd in the tech corridor looks 
less San Francisco—young white guys in hoodies—and more 
international. There’s free health care and good public schools, 

Maybe we’re not 
actually at the 
beginning of a 
revolution. 

the people are friendly, and the political order is relatively left- 
leaning and stable; and this stuff draws people like Hinton, 
who says he left the U.S. because of the Iran-Contra affair. It’s 
one of the first things we talk about when I go to meet him, 
just before lunch. 

“Most people at CMU thought it was perfectly reasonable 
for the U.S. to invade Nicaragua,” he says. “They somehow 
thought they owned it.” He tells me that he had a big break¬ 
through recently on a project: “getting a very good junior engi¬ 
neer who’s working with me,” a woman named Sara Sabour. 
Sabour is Iranian, and she was refused a visa to work in the 
United States. Google’s Toronto office scooped her up. 

Hinton, who is 69 years old, has the kind, lean, English- 
looking face of the Big Friendly Giant, with a thin mouth, big 
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ears, and a proud nose. He was born in Wimbledon, England, 
and sounds, when he talks, like the narrator of a children’s 
book about science: curious, engaging, eager to explain things. 
He’s funny, and a bit of a showman. He stands the whole time 
we talk, because, as it turns out, sitting is too painful. “I sat 
down in June of 2005 and it was a mistake,” he tells me, letting 
the bizarre line land before explaining that a disc in his back 
gives him trouble. It means he can’t fly, and earlier that day 
he’d had to bring a contraption that looked like a surfboard to 
the dentist’s office so he could lie on it while having a cracked 
tooth root examined. 

In the 1980s Hinton was, as he is now, an expert on neu¬ 
ral networks, a much-simplified model of the network of neu¬ 
rons and synapses in our brains. However, at that time it had 
been firmly decided that neural networks were a dead end in 
AI research. Although the earliest neural net, the Perceptron, 
which began to be developed in the 1950s, had been hailed as 
a first step toward human-level machine intelligence, a 1969 
book by MIT’s Marvin Minsky and Seymour Papert, called Per- 
ceptrons, proved mathematically that such networks could per¬ 
form only the most basic functions. These networks had just 
two layers of neurons, an input layer and an output layer. Nets 
with more layers between the input and output neurons could 
in theory solve a great variety of problems, but nobody knew 
how to train them, and so in practice they were useless. Except 
for a few holdouts like Hinton, Perceptrons caused most people 
to give up on neural nets entirely. 

Hinton’s breakthrough, in 1986, was to show that back- 
propagation could train a deep neural net, meaning one with 
more than two or three layers. But it took another 26 years 
before increasing computational power made good on the 
discovery. A 2012 paper by Hinton and two of his Toronto 
students showed that deep neural nets, trained using back- 
propagation, beat state-of-the-art systems in image recogni¬ 
tion. “Deep learning” took off. To the outside world, AI seemed 
to wake up overnight. For Hinton, it was a payoff long overdue. 

Reality distortion field 

A neural net is usually drawn like a club sandwich, with lay¬ 
ers stacked one atop the other. The layers contain artificial 
neurons, which are dumb little computational units that get 
excited—the way a real neuron gets excited—and pass that 
excitement on to the other neurons they’re connected to. A 
neuron’s excitement is represented by a number, like 0.13 or 
32.39, that says just how excited it is. And there’s another cru¬ 
cial number, on each of the connections between two neurons, 
that determines how much excitement should get passed from 
one to the other. That number is meant to model the strength 
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A diagram from seminal work on 
“error propagation” by Hinton, David 
Rumelhart, and Ronald Williams. 


of the synapses between neurons in the brain. When the num¬ 
ber is higher, it means the connection is stronger, so more of 
the one’s excitement flows to the other. 

One of the most successful applications of deep neural nets 
is in image recognition—as in the memorable scene in HBO’s 
Silicon Valley where the team builds a program that can tell 
whether there’s a hot dog in a picture. Programs like that actu¬ 
ally exist, and they wouldn’t have been possible a decade ago. 
To get them to work, the first step is to get a picture. Let’s say, 
for simplicity, it’s a small black-and-white image that’s 100 
pixels wide and 100 pixels tall. You feed this image to your 
neural net by setting the excitement of each simulated neuron 
in the input layer so that it’s equal to the brightness of each 
pixel. That’s the bottom layer of the club sandwich: 10,000 
neurons (100x100) representing the brightness of every pixel 
in the image. 

You then connect this big layer of neurons to another big 
layer of neurons above it, say a few thousand, and these in turn 
to another layer of another few thousand neurons, and so on 
for a few layers. Finally, in the topmost layer of the sandwich, 
the output layer, you have just two neurons—one representing 
“hot dog” and the other representing “not hot dog.” The idea is 
to teach the neural net to excite only the first of those neurons 
if there’s a hot dog in the picture, and only the second if there 
isn’t. Backpropagation—the technique that Hinton has built 
his career upon—is the method for doing this. 

Backprop is remarkably simple, though it works best with 
huge amounts of data. That’s why big data is so important in 
AI—why Facebook and Google are so hungry for it, and why 
the Vector Institute decided to set up shop down the street 
from four of Canada’s largest hospitals and develop data part¬ 
nerships with them. 

In this case, the data takes the form of millions of pic¬ 
tures, some with hot dogs and some without; the trick is that 
these pictures are labeled as to which have hot dogs. When you 
first create your neural net, the connections between neurons 
might have random weights—random numbers that say how 
much excitement to pass along each connection. It’s as if the 
synapses of the brain haven’t been tuned yet. The goal of back- 
prop is to change those weights so that they make the network 
work: so that when you pass in an image of a hot dog to the 
lowest layer, the topmost layer’s “hot dog” neuron ends up get¬ 
ting excited. 

Suppose you take your first training image, and it’s a 
picture of a piano. You convert the pixel intensities of the 
100x100 picture into 10,000 numbers, one for each neuron in 
the bottom layer of the network. As the excitement spreads up 
the network according to the connection strengths between 
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neurons in adjacent layers, it’ll eventually end up in that last 
layer, the one with the two neurons that say whether there’s 
a hot dog in the picture. Since the picture is of a piano, ide¬ 
ally the “hot dog” neuron should have a zero on it, while the 
“not hot dog” neuron should have a high number. But let’s say 
it doesn’t work out that way. Let’s say the network is wrong 
about this picture. Backprop is a procedure for rejiggering the 
strength of every connection in the network so as to fix the 
error for a given training example. The way it works is that you 
start with the last two neurons, and figure out just how wrong 
they were: how much of a difference is there between what the 
excitement numbers should have been and what they actually 
were? When that’s done, you take a look at each of the connec¬ 
tions leading into those neurons—the ones in the next lower 
layer—and figure out their contribution to the error. You keep 
doing this until you’ve gone all the way to the first set of con¬ 
nections, at the very bottom of the network. At that point you 

A real intelligence 
doesn't break when 
you slightly change 
the problem. 

know how much each individual connection contributed to 
the overall error, and in a final step, you change each of the 
weights in the direction that best reduces the error overall. 

The technique is called “backpropagation” because you are 
“propagating” errors back (or down) through the network, 
starting from the output. 

The incredible thing is that when you do this with millions 
or billions of images, the network starts to get pretty good at 
saying whether an image has a hot dog in it. And what’s even 
more remarkable is that the individual layers of these image- 
recognition nets start being able to “see” images in sort of the 
same way our own visual system does. That is, the first layer 
might end up detecting edges, in the sense that its neurons 
get excited when there are edges and don’t get excited when 
there aren’t; the layer above that one might be able to detect 
sets of edges, like corners; the layer above that one might start 
to see shapes; and the layer above that one might start find¬ 
ing stuff like “open bun” or “closed bun,” in the sense of having 


neurons that respond to either case. The net organizes itself, in 
other words, into hierarchical layers without ever having been 
explicitly programmed that way. 

This is the thing that has everybody enthralled. It’s not just 
that neural nets are good at classifying pictures of hot dogs 
or whatever: they seem able to build representations of ideas. 
With text you can see this even more clearly. You can feed the 
text of Wikipedia, many billions of words long, into a simple 
neural net, training it to spit out, for each word, a big list of 
numbers that correspond to the excitement of each neuron in 
a layer. If you think of each of these numbers as a coordinate in 
a complex space, then essentially what you’re doing is finding 
a point, known in this context as a vector, for each word some¬ 
where in that space. Now, train your network in such a way that 
words appearing near one another on Wikipedia pages end up 
with similar coordinates, and voila, something crazy happens: 
words that have similar meanings start showing up near one 
another in the space. That is, “insane” and “unhinged” will have 
coordinates close to each other, as will “three” and “seven,” and 
so on. What’s more, so-called vector arithmetic makes it pos¬ 
sible to, say, subtract the vector for “France” from the vector for 
“Paris,” add the vector for “Italy,” and end up in the neighbor¬ 
hood of “Rome.” It works without anyone telling the network 
explicitly that Rome is to Italy as Paris is to France. 

“It’s amazing,” Hinton says. “It’s shocking.” Neural nets can 
be thought of as trying to take things—images, words, record¬ 
ings of someone talking, medical data—and put them into 
what mathematicians call a high-dimensional vector space, 
where the closeness or distance of the things reflects some 
important feature of the actual world. Hinton believes this is 
what the brain itself does. “If you want to know what a thought 
is,” he says, “I can express it for you in a string of words. I 
can say ‘John thought, “Whoops.”’ But if you ask, ‘What is the 
thought? What does it mean for John to have that thought?’ 

It’s not that inside his head there’s an opening quote, and a 
‘Whoops,’ and a closing quote, or even a cleaned-up version of 
that. Inside his head there’s some big pattern of neural activ¬ 
ity.” Big patterns of neural activity, if you’re a mathematician, 
can be captured in a vector space, with each neuron’s activity 
corresponding to a number, and each number to a coordinate 
of a really big vector. In Hinton’s view, that’s what thought is: a 
dance of vectors. 

It is no coincidence that Toronto’s flagship AI institution 
was named for this fact. Hinton was the one who came up with 
the name Vector Institute. 

There’s a sort of reality distortion field that Hinton cre¬ 
ates, an air of certainty and enthusiasm, that gives you the 
feeling there’s nothing that vectors can’t do. After all, look 
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at what they’ve been able to produce already: cars that drive 
themselves, computers that detect cancer, machines that 
instantly translate spoken language. And look at this charm¬ 
ing British scientist talking about gradient descent in high¬ 
dimensional spaces! 

It’s only when you leave the room that you remember: 
these “deep learning” systems are still pretty dumb, in spite 
of how smart they sometimes seem. A computer that sees a 
picture of a pile of doughnuts piled up on a table and cap¬ 
tions it, automatically, as “a pile of doughnuts piled on a table” 
seems to understand the world; but when that same program 
sees a picture of a girl brushing her teeth and says “The boy is 
holding a baseball bat,” you realize how thin that understand¬ 
ing really is, if ever it was there at all. Neural nets are just 
thoughtless fuzzy pattern recognizers, and as useful as fuzzy 
pattern recognizers can be—hence the rush to integrate them 
into just about every kind of software—they represent, at 
best, a limited brand of intelligence, one that is easily fooled. 

A deep neural net that recognizes images can be totally sty¬ 
mied when you change a single pixel, or add visual noise that’s 
imperceptible to a human. Indeed, almost as often as we’re 
finding new ways to apply deep learning, we’re finding more 
of its limits. Self-driving cars can fail to navigate conditions 
they’ve never seen before. Machines have trouble parsing sen¬ 
tences that demand common-sense understanding of how the 
world works. 

Deep learning in some ways mimics what goes on in the 
human brain, but only in a shallow way—which perhaps 
explains why its intelligence can sometimes seem so shallow. 
Indeed, backprop wasn’t discovered by probing deep into the 
brain, decoding thought itself; it grew out of models of how 
animals learn by trial and error in old classical-conditioning 
experiments. And most of the big leaps that came about as it 
developed didn’t involve some new insight about neuroscience; 
they were technical improvements, reached by years of math¬ 
ematics and engineering. What we know about intelligence is 
nothing against the vastness of what we still don’t know. 

David Duvenaud, an assistant professor in the same 
department as Hinton at the University of Toronto, says deep 
learning has been somewhat like engineering before physics. 
“Someone writes a paper and says, 1 made this bridge and it 
stood up!’ Another guy has a paper: 1 made this bridge and 
it fell down—but then I added pillars, and then it stayed up.’ 
Then pillars are a hot new thing. Someone comes up with 
arches, and it’s like, Arches are great!”’ With physics, he says, 
“you can actually understand what’s going to work and why.” 
Only recently, he says, have we begun to move into that phase 
of actual understanding with artificial intelligence. 
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Hinton himself says, “Most conferences consist of making 
minor variations ... as opposed to thinking hard and saying, 
What is it about what we’re doing now that’s really deficient? 
What does it have difficulty with? Let’s focus on that.’” 

It can be hard to appreciate this from the outside, when 
all you see is one great advance touted after another. But the 
latest sweep of progress in AI has been less science than engi¬ 
neering, even tinkering. And though we’ve started to get a 
better handle on what kinds of changes will improve deep¬ 
learning systems, we’re still largely in the dark about how those 
systems work, or whether they could ever add up to something 
as powerful as the human mind. 

It’s worth asking whether we’ve wrung nearly all we can 
out of backprop. If so, that might mean a plateau for progress 
in artificial intelligence. 

Patience 

If you want to see the next big thing, something that could 
form the basis of machines with a much more flexible intel¬ 
ligence, you should probably check out research that resem¬ 
bles what you would’ve found had you encountered backprop 
in the ’80s: smart people plugging away on ideas that don’t 
really work yet. 

A few months ago I went to the Center for Minds, Brains, 
and Machines, a multi-institutional effort headquartered at 
MIT, to watch a friend of mine, Eyal Dechter, defend his dis¬ 
sertation in cognitive science. Just before the talk started, his 
wife Amy, their dog Ruby, and their daughter Susannah were 
milling around, wishing him well. On the screen was a pic¬ 
ture of Ruby, and next to it one of Susannah as a baby. When 
Dad asked Susannah to point herself out, she happily slapped 
a long retractable pointer against her own baby picture. On 
the way out of the room, she wheeled a toy stroller behind 
her mom and yelled “Good luck, Daddy!” over her shoulder. 
“Vamanos!” she said finally. She’s two. 

Eyal started his talk with a beguiling question: How is it 
that Susannah, after two years of experience, can learn to talk, 
to play, to follow stories? What is it about the human brain 
that makes it learn so well? Will a computer ever be able to 
learn so quickly and so fluidly? 

We make sense of new phenomena in terms of things we 
already understand. We break a domain down into pieces and 
learn the pieces. Eyal is a mathematician and computer pro¬ 
grammer, and he thinks about tasks—like making a souffle—as 
really complex computer programs. But it’s not as if you learn 
to make a souffle by learning every one of the program’s zillion 
micro-instructions, like “Rotate your elbow 30 degrees, then 
look down at the countertop, then extend your pointer finger, 
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Hinton made this sketch for 
his next big idea, to organize 
neural nets with “capsules.” 
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then ...” If you had to do that for every new task, learning 
would be too hard, and you’d be stuck with what you already 
know. Instead, we cast the program in terms of high-level 
steps, like “Whip the egg whites,” which are themselves com¬ 
posed of subprograms, like “Crack the eggs” and “Separate 
out the yolks.” 

Computers don’t do this, and that is a big part of the reason 
they’re dumb. To get a deep-learning system to recognize a hot 
dog, you might have to feed it 40 million pictures of hot dogs. 
To get Susannah to recognize a hot dog, you show her a hot 
dog. And before long she’ll have an understanding of language 
that goes deeper than recognizing that certain words often 
appear together. Unlike a computer, she’ll have a model in her 
mind about how the whole world works. “It’s sort of incredible 
to me that people are scared of computers taking jobs,” Eyal 
says. “It’s not that computers can’t replace lawyers because 
lawyers do really complicated things. It’s because lawyers read 
and talk to people. It’s not like we’re close. We’re so far.” 

A real intelligence doesn’t break when you slightly change 
the requirements of the problem it’s trying to solve. And the 
key part of Eyal’s thesis was his demonstration, in principle, 
of how you might get a computer to work that way: to fluidly 
apply what it already knows to new tasks, to quickly bootstrap 
its way from knowing almost nothing about a new domain to 
being an expert. 

Essentially, it is a procedure he calls the “exploration- 
compression” algorithm. It gets a computer to function some¬ 
what like a programmer who builds up a library of reusable, 
modular components on the way to building more and more 
complex programs. Without being told anything about a new 
domain, the computer tries to structure knowledge about it 
just by playing around, consolidating what it’s found, and 
playing around some more, the way a human child does. 

His advisor, Joshua Tenenbaum, is one of the most highly 
cited researchers in AI. Tenenbaum’s name came up in half 
the conversations I had with other scientists. Some of the 
key people at DeepMind—the team behind AlphaGo, which 
shocked computer scientists by beating a world champion 
player in the complex game of Go in 2016—had worked as 
his postdocs. He’s involved with a startup that’s trying to give 
self-driving cars some intuition about basic physics and other 
drivers’ intentions, so they can better anticipate what would 
happen in a situation they’ve never seen before, like when a 
truck jackknifes in front of them or when someone tries to 
merge very aggressively. 

Eyal’s thesis doesn’t yet translate into those kinds of prac¬ 
tical applications, let alone any programs that would make 
headlines for besting a human. The problems Eyal’s working 


on “are just really, really hard,” Tenenbaum said. “It’s gonna 
take many, many generations.” 

Tenenbaum has long, curly, whitening hair, and when we 
sat down for coffee he had on a button-down shirt with black 
slacks. He told me he looks to the story of backprop for inspi¬ 
ration. For decades, backprop was cool math that didn’t really 
accomplish anything. As computers got faster and the engi¬ 
neering got more sophisticated, suddenly it did. He hopes the 
same thing might happen with his own work and that of his 
students, “but it might take another couple decades.” 

As for Hinton, he is convinced that overcoming AI’s limi¬ 
tations involves building “a bridge between computer science 
and biology.” Backprop was, in this view, a triumph of biologi¬ 
cally inspired computation; the idea initially came not from 
engineering but from psychology. So now Hinton is trying to 
pull off a similar trick. 

“The fact that it 
doesn’t work is 
just a temporary 
annoyance.” 

Neural networks today are made of big flat layers, but in 
the human neocortex real neurons are arranged not just hori¬ 
zontally into layers but vertically into columns. Hinton thinks 
he knows what the columns are for—in vision, for instance, 
they’re crucial for our ability to recognize objects even as our 
viewpoint changes. So he’s building an artificial version— 
he calls them “capsules”—to test the theory. So far, it hasn’t 
panned out; the capsules haven’t dramatically improved his 
nets’ performance. But this was the same situation he’d been in 
with backprop for nearly 30 years. 

“This thing just has to be right,” he says about the cap¬ 
sule theory, laughing at his own boldness. “And the fact that it 
doesn’t work is just a temporary annoyance.” D 


James Somers is a writer and programmer based in New 
York City. His previous article for MIT Technology Review 
was “Toolkits for the Mind” in May/June 2015, which showed 
how Internet startups are shaped by the programming 
languages they use. 
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Fearsome Machines: 
A Prehistory 

Ever since the beginning of industrial society, people have 
simultaneously marveled at the power of automation and 
lamented that human capabilities are being irredeemably 
devalued. 


► 1811- Demanding better conditions 
and higher pay, textile workers in 
England smash machinery and set 
factories on fire. These workers will 
come to be known as Luddites, after 
their mythical leader, Ned Ludd, and 
the name will become a synonym for 
opponents or critics of technology. 
But it’s a misnomer: this is a class pro¬ 
test more than a technological one. 
The stocking-frame machines the 
Luddites vandalize have been around 
since the 1600s. 

► 1829 -Grappling with the meaning 
of the “mechanical age,” the philos¬ 
opher Thomas Carlyle notes that 
industry has made society richer, 
and that people are generally better 
“fed, clothed, lodged, and, in all out¬ 
ward respects, accommodated.” But 
he questions an increasing “distance 
between the rich and the poor” and 
notes a “mighty change” in “modes of 
thought and feeling”: “Men are grown 
mechanical in head and in heart, as 
well as in hand.” Will civic life lose its 
soul? After all, “government includes 
much also that is not mechanical, and 
cannot be treated mechanically.” 

► 1842 - Ada Lovelace envisions how a 
calculating machine “might act upon 
other things besides number[s]”— 
for example, composing music. She 


also shows how the machine could be 
made to determine “that which human 
brains find it difficult or impossible to 
work out unerringly.” 

► 1868 -“Calculation and reasoning, 
like weaving and plowing, are work, 
not for human souls, but for clever 
combinations of iron and wood. If 
you spend time in doing work that a 
machine could do faster than your¬ 
selves, it should only be for exercise.” 

— Mary Everest Boole 



► 1917 - “The machine is the very image 
of efficiency—governed, deadly, pre- 
destinarian. But it is an efficiency 
against which nature cries out; it is 
an efficiency destitute of that adapt¬ 
ability of means and idealization of 
ends which is the human essence of 
true reason.” —Hartley Burr Alexander 


f’ 1955- Mathematics professor John 
McCarthy calls for researchers to 
join him the following summer at 
Dartmouth College to build the foun¬ 
dation for artificial intelligence. He 
says, “Every aspect of learning or any 
other feature of intelligence can in 
principle be so precisely described 
that a machine can be made to simu¬ 
late it.” 

► 1977 - McCarthy predicts that bring¬ 
ing Al to the level of humans will take 
“five to 500 years.” 

Y 2009 -“The economic problems of 
the future will not be about growth 
but about something more nettle- 
some: the ineluctable increase in the 
number of people with no marketable 
skills, and technology’s role not as 
the antidote to social conflict, but as 
its instigator.” —Gregory Clark 

Y 2014-“l think we should be very 
careful about artificial intelligence. 
If I were to guess what our biggest 
existential threat is, it’s probably that. 
With artificial intelligence we are 
summoning the demon.” 

—Elon Musk 

► 2016 -After studying the economic 
effects of advances in Al, the Obama 
White House suggests increases in 
the minimum wage, new training pro¬ 
grams, and policies aimed at helping 
workers adjust. But it says the more 
aggressive step of enacting a univer¬ 
sal basic income could be “counter¬ 
productive”: “We should not advance 
a policy that is premised on giving up 
on the possibility of workers’ remain¬ 
ing employed.” 

} 2017 -Russian president Vladimir 
Putin says that whichever country 
leads the development of Al will be 
“the ruler of the world.” 
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Two days after K.S. Sunil Kumar received a promotion, 
Human Resources phoned him up and asked him to resign. 

This happened in April, just as Kumar was beginning his 
ninth year at Tech Mahindra, one of the giants in India’s IT 
services industry. He worked in engineering services, where 
he designed components and tools for aerospace firms in 
North America and Europe. They’d send over specs—the 
materials available to construct a hinge, and the kind of load 
it had to bear, and the cost at which it had to be manufac¬ 
tured—and he mocked up options with the help of software. 
He was a foot soldier in the army of Indian engineers to 
whom work is outsourced from the West, so that it can be fin¬ 
ished at a fraction of the expense. Sometimes he left his base, 
Tech Mahindra’s Bangalore campus, to serve stints at clients’ 
offices abroad: in Montreal, Belfast, or Stockholm. When his 
employment was terminated, Kumar was earning close to 
$ 17,000 annually, a nice middle-class salary in India. Around 
the same time, Tech Mahindra announced profits for the 
previous financial year of $419 million, on revenue of $4.35 
billion. (Tech Mahindra did not respond to a request for com¬ 
ment for this story.) IT services and related offerings in India 
record annual revenue of $154 billion and employ nearly four 
million people. The sector’s vigor has relied upon its ability to 
shave costs lower and lower—upon its ability to arbitrage the 
cheaply bought skills of workers like Sunil Kumar. 

Bangalore is filled with IT professionals and engineers 
who resemble him. His curly hair is thin on top and gray at 
the temples; when we spoke he was wearing a faded checked 
Tommy Hilfiger shirt, a backpack, and a look of subdued 
anxiety. He grew up in a village a few hundred miles from 
Bangalore, where his father wove silk saris on a handloom. 

In 1995, when he was 15, he moved to Bangalore to study for 
a diploma in mechanical engineering; it was a step below a 
university degree, which he would gain later through a cor¬ 
respondence course. Until he joined Tech Mahindra, in the 
summer of 2008, Kumar worked as a draftsman at an aero¬ 
space firm. The new job opened his life up the way the IT 
industry did for so many Indians, offering a chance to vault 
from a blue-collar background to a white-collar future. He 
got married, and the couple had a son; he took out a loan of 
$47,000 to buy a house, so that his parents and two brothers, 
who followed him to Bangalore, could stay with him. “I live a 
middle-class life,” he says. “I don’t want to showcase to people 
that I’m doing an IT job. Branded shirts, shoes—I don’t want 
them.” 

When Kumar lost his job, he became part of a wave of 
layoffs washing through the Indian IT industry—a term that 
includes, in its vastness, call centers, engineering services, 
business process outsourcing firms, and infrastructure man¬ 
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agement and software companies. The recent layoffs are part 
of the industry’s most significant period of churn since it 
began to boom two decades ago. Companies don’t necessar¬ 
ily attribute these layoffs directly to automation, but at the 
same time, they constantly identify automation as the spark 
for huge changes in the industry. Bots, machine learning, and 
algorithms that robotically execute processes are rendering 
old skills redundant, recasting the idea of work and making 
a smaller labor force seem likely. An analysis by the busi¬ 
ness newspaper Mint reported that the top seven IT firms in 
India will lay off at least 56,000 employees this year. After its 
annual general meeting this summer, the $10 billion behe¬ 
moth Infosys announced that 11,000 of its 200,000 employ¬ 
ees had been “released” from repetitive tasks by automation 
and redeployed elsewhere within the company, the burden of 
their previous work now borne by algorithms. HfS Research, 
which studies the IT industry, predicted last year that auto¬ 
mation will result in a net loss of 480,000 jobs in India by 
2021. “If we sit still, there is absolutely no doubt that our jobs 
are going to be wiped out by AI,” Vishal Sikka said in March, 
when he was still the CEO of Infosys. (Sikka resigned in 
August.) “Sixty to 70 percent over the next 10 years—or maybe 
less than 10 years—of the jobs that we do today are going to 
be replaced by AI... unless we continue to evolve ourselves.” 


Ultra-Fast Growth 

The IT and business process outsourcing industry has 
become the largest private-sector employer in India. 
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The fear that AI is pulverizing jobs is not unique to India, 
but automation could hit this country particularly hard 
because so much of its high-tech economy involves relatively 
routine work that is prime for computers to take over. In 
some cases, Indian IT services companies will automate the 
work themselves. In other cases, companies in the West will 
do it, so they no longer have to farm work out to humans in 
India. 

Sunil Kumar wasn’t told, in any detail, why he was being 
let go; he insists that what he did at Tech Mahindra wasn’t 
automatable, and that he was fired as part of a drive to invig¬ 
orate the bottom line. Devika Narayan, a sociologist at the 
University of Minnesota who is researching the subject for 
her doctoral thesis, thinks automation may indeed be getting 
too much blame for the loss of jobs like his. Companies might 
well be talking up automation to mask some of their own fail¬ 
ures, or to distract from the ill effects of other factors beyond 
their control, she says. She points out that many IT giants are 
flabby and overstaffed, and that American companies are now 
wary, given the U.S. political climate, of sending work over¬ 
seas. “The extent to which automation is being exaggerated 
is still unclear to me,” Narayan says. She suspects that Indian 
IT companies “want to leverage this automation narrative to 
undertake structural changes, particularly downsizing.’” 

Where the truth lies is important for India. The IT indus¬ 
try may employ only a few million of India’s 1.3 billion peo¬ 
ple—but it has been a beacon for young men and women 
with aspirations. It motivated families to send their children 
to university, placed graduates in gleaming campuses, con¬ 
ferred independent urban lifestyles upon them, and provided 
stable incomes and access to the world outside India. Over 
the last 30 years, moreover, it has been the only industry in 
India to begin from seed and bloom to such success. India is 
otherwise struggling to create jobs: 12 million Indians enter 
the workforce every year, but only 135,000 jobs in the formal 
economy’s eight biggest sectors—including IT—were created 
in 2015. A dramatic contraction of the IT industry—a dim¬ 
ming of the beacon—would jolt the country’s economy and 
polity deeply. 

Taking out the swivel chairs 

Chetan Dube says he saw it coming. In 2005, Dube, the CEO 
of a company called IPsoft, was addressing a forum of IT com¬ 
panies in Mumbai. “If Indian industry does not wake up to the 
automation wave that is coming, then we will face an existential 
crisis,” he remembers telling the gathering. “I got chastised. The 
next day, we were having breakfast, and at the back of the Eco¬ 
nomic Times , an article said: IPsoft CEO predicts the death of 
Indian outsourcing.’” 


Dube, a bow-tied and suspendered mathematician who once 
taught at New York University, founded IPsoft in 1998, but it 
was in 2014 that the company launched what is now its flag¬ 
ship product, Amelia. A conversational service agent, Amelia is 
designed to replace the humans who field customer queries in 
call centers and back offices. Amelia has been used to resolve 
suppliers’ questions for a large oil and gas firm; it runs the live 

Automation could hit 
India particularly hard 
because much of its 
high-tech economy 
involves relatively 
routine work that is 
prime for computers 
to take over. 

chat service for SEB, a Swedish bank; it works in another bank’s 
team of mortgage brokers. For one client, Dube says, the average 
time to reach a conventional offshore agent used to be 55 sec¬ 
onds; an incarnation of Amelia made itself available in two sec¬ 
onds or less. An offshore agent needed 18.2 minutes, on average, 
to resolve a query successfully; for Amelia, it was 4.5 minutes. 
The customer service sphere is filling rapidly with products like 
Amelia—chatbots that, through text or voice, obviate the human 
presence altogether. 

Only in a few cases has Amelia directly supplanted Indian 
workers, but Dube thinks that further change is inevitable. 

Call centers in India are already in flux: salaries have crept up, 
attrition has always been high, and firms like Infosys and Tata 
Consultancy Services have offshored some of their functions 
farther out to Manila, where labor costs are even lower than in 
India. (Three years ago, one official in an industry body named 
Assocham predicted gloomily that India would lose $30 bil¬ 
lion in call center revenues to the Philippines over the follow¬ 
ing decade.) In the West, some companies are repatriating their 
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voice service operations, while others are abandoning them alto¬ 
gether in favor of e-mail or chat help desks. 

The prospect—or fear—of automation has thus become one 
more force reshaping the call center business. Voice recognition 
isn’t anywhere close to perfect yet, and even the sophisticated 
cognitive agents of the vaunted near future may not be able to 
parse rambling customers, complicated problems, or uncom¬ 
monly thick accents. But most voice work is prosaic and repeti¬ 
tive. Given that humans in the first tier of this service calibrate 
their responses with the help of a script, their functions are 
among the simplest to transform into machine code. 

Other fruit hangs similarly low elsewhere in the sector; as 
Dube says, “India is nothing but the blue-collar worker of IT,” so 
the lowest layer of work is plump with tasks that need diligence 
and stamina but not creativity or sharp technical skill. 

At Genpact, a 20-year-old company in the industry of busi¬ 
ness process outsourcing, there’s a lot of “swivel-chair work,” 
says Gianni Giacomelli, who leads the firm’s digital solutions 
business. The term captures the mechanical nature of these 
tasks. Until recently, a human has been required to deal with 
software systems that help with enterprise functions. Those 
systems are often disconnected from each other, so Genpact’s 
employees “are asked to process, very basically, things that come 
from one system and go into another system,” he says. “That 
toggling back and forth is a massive waste of time.” Since 2014, 
Genpact has been replacing workers in swivel chairs by order¬ 
ing computers to take information from screens and servers and 
convey it into another system. 

One level up is the kind of work that Giacomelli calls “rec¬ 
onciliation”: examining invoices and bills from a client’s various 
vendors and customers, with all their discrepancies and contra¬ 
dictions. It isn’t trivial work; it involves, right now, some grains 
of human judgment. “But once machines have seen enough of 
those things, they can do that kind of stuff,” he says. 

Mayhem 

For some of its clients, the IT colossus Infosys has been able to 
automate nearly all of the most routine chores of monitoring and 
maintaining their data infrastructure, says S. Ravi Kumar, the 
company’s deputy COO. Some intermediate work, such as triag¬ 
ing IT service requests, is now done by machines as well. At a 
still-higher level of service complexity—jobs that involve trouble¬ 
shooting bugs deep within the code, or developing solutions to 
new problems—35 to 40 percent of tasks are performed by auto¬ 
mated routines. 

Overall, Somak Roy, an analyst at Forrester Research, esti¬ 
mates that only a quarter of the most easily automated work in 
India is being completed exclusively by machines. Companies 
are still enthusiastically dabbling in technologies that remain 


nascent. Nonetheless, Roy calls it a “distinct possibility” that IT 
will “cease to be a large-scale employment generator in India.” 

One of the direst visions comes from Pankaj Bansal, the 
chief executive of PeopleStrong, a human-resources firm that 
frequently staffs IT companies with engineers. For IT services 
firms in the shape and form India has known them, Bansal says, 

“it will be mayhem.” He has been accused of fearmongering, but 
he holds fast to his assessment. Over the last two years, three or 
four out of every 10 jobs in the bottom layer of the pyramid of 

Bansal reckons that 
the IT sector hired 
400,000 people 
annually until two 
or three years ago, 
and that the number 
has now shrunk to 
140,000 to 160,000. 
Soon, he says, “net 
hiring will be barely 
above zero.” 

IT work have been “squashed” by automation, he says—and this 
has been manifested not in how many people have been laid off 

but in how sharply recruitment has dropped. Companies once | 

< 

gusted through the campuses of engineering colleges, picking 
them clean of fresh graduates. Bansal reckons that the IT sector | 

hired 400,000 people annually until two or three years ago, and 
that the number has now shrunk to 140,000 to 160,000. Soon, he 8 
says, “net hiring will be barely above zero.” 1 

Bansal’s prophesy of a deflating workforce may well come | 

true for another reason. For years, IT firms hired inexpensive, 
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The size and prestige of India’s IT industry are reflected in the 
design of Infosys campuses in Mysore (top) and Bangalore. 
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hardworking youngsters wholesale—even if they were poorly 
skilled—because it made sense to staff projects heavily. The more 
warm bodies assigned to a task, the higher the bill that could be 
presented to the client. But the practice of calculating bills in this 
manner has waned; customers now pay for outcomes and impact. 
Meanwhile, the poorly skilled youngsters who stayed with their 
firms have received promotions and raises with clockwork regu¬ 
larity, until they’ve turned into mid-rung engineers who are now 
too costly, in their thousands, to sustain. Cue the purges. 

Within the industry, Bansal’s grim views encounter profound 
disagreement, at least in public. Perhaps this is understandable: 
it has never been wise for companies to be effusive about the 
imminence of layoffs and workforce reductions. Sangeeta Gupta, 
a senior vice president at an industry body called the National 
Association of Software and Services Companies, predicts only a 
“decoupling” of revenues and head count over the next few years. 
If Indian IT required three million employees to touch $100 bil¬ 
lion in annual revenue, she says, it will need only 1.2 million to 
two million additional people for its next $100 billion. By 2025, 
when revenues reach $350 billion, Gupta predicts, the sector will 
have added another 2.5 million to three million jobs to the four 
million it holds today. 

There is a tension 
between the long 
arc of technological 
revolutions and the 
far shorter one of 
human lives. 

Companies are eager to explain why automation won’t hollow 
out, and might even expand, their swarms of employees. For one 
thing, it isn’t as if machines can make people instantly redundant. 
“Jobs are not structured in such a clean way,” says Giacomelli, at 
Genpact. The architectures of modem work that have developed 
over decades all have human beings at their center; they rely upon 
people’s agility and their capacity to hold different things in their 
minds. “People do many things, so it’s not that easy to extricate 
one task or the other and make that happen through AI,” he says. 


Companies also insist that they want to reskill the employees 
who risk being supplanted by automation. If an engineer’s work 
is best taken over by an algorithm, “it isn’t fair to then say You 
don’t have a job,”’ says K.M. Madhusudhan, the CTO of Mind- 
tree, a services firm that employs more than 16,000 people. 

“Can we teach this engineer programming? Maybe not heavy 
lifting, but some scripting, which is not that difficult. For every 
role, we believe, there are adjacent, higher-level skills that can 
be acquired.” Madhusudhan calls this a “humane approach.” It 
will result in fewer job losses, although he acknowledges that 
firms like his will also be creating fewer jobs. “The numbers that 
were possible before will not be possible in the future,” he says. 
“That is the bigger concern for a country like India, because we 
still produce a lot of engineers, and not everyone will get a job.” 

This is a familiar pattern in history: every technological 
stride forward has meant that the same amount of work can be 
done by fewer people. “Whenever there’s a revolution, there’s a 
worry about fewer jobs. It happened with the Industrial Revolu¬ 
tion as well,” says Ravi Kumar, at Infosys. “The reality is, though, 
that there’s more consumption,” he adds. That eventually 
increases the need for new kinds of labor. At the moment, he 
says, enterprises spend 65 to 70 percent of their IT budgets “just 
to keep the lights on”—to pay for infrastructure and routine sup¬ 
port. If that money is undammed, it may well pour into new— 
and as yet unimagined—streams of revenue and employment: 

“It would mean a much bigger canvas for us.” 

But even if he is right, there is a tension between the long arc 
of these revolutions and the far shorter one of human lives. In 
the near term, people will lose their livelihoods. Sunil Kumar is 
still without a job. 

In June, he filed a petition for wrongful dismissal with the 
office of the labor commissioner, a state body that resolves 
industrial disputes and enforces labor laws. Once, when he 
checked on its progress, an official advised him that his fight 
was likely to be a lengthy one, and now he suspects nothing will 
come of it. “Whatever confidence I had, I’m losing it,” he says. 
When he reads his newspapers, he stops just short of the busi¬ 
ness pages, which frustrate him. “There will be companies say¬ 
ing many things: We’re hiring this many people, there are many 
opportunities.’ The CEOs keep saying it. I stopped reading all 
this,” he says. He knows he ought to start looking for a new job, 
but he hasn’t been able to pull himself together; it is as if his dis¬ 
missal had stymied life itself. “I haven’t been able to concentrate 
on anything,” he says. “It’s very difficult now.” H 


Samanth Subramanian has written for such publications as 
Wired, the New York Times Magazine, and the New Yorker. His 
most recent book is This Divided Island: Life, Death, and the Sri 
Lankan War. 
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The Dangers of 
Tech-BroAl 

Tabitha Goldstaub, a cofounder of CognitionX, which 
helps companies deploy Al, says that diversifying the field 
is necessary to make sure products actually work well. 


If you think about artificial intelligence and the core 
components of what makes an Al, you have the fact 
that it’s given a goal and then will find a way to reach 
that goal. It’s then often very un-transparent as to how 
it reached that goal. If you’re building into a machine 
some unconscious bias, you might not know that 
it’s there; the output could be detrimental to women 
and it’s very tough to work out exactly why that has 
happened. 

In traditional technology, you can see [what has 
happened]: women dying in car crashes because the 
crash-test dummies were the shape of a man rather 
than the shape of a woman. [With Al there could be] 
similar life-or-death situations, in drug trials or in 
autonomous vehicles and things like that. 

There are some examples of [gender bias in Al 
today]: Google ads displaying higher-earning [job] 
ads to men than women. We can hypothesize other 
types of situations that would happen—what if 
women weren’t as able to get loans or mortgages 
or insurance? 

I don’t have a dystopian view of Al. I don’t see 
killer robots. I’m so much more focused on the narrow 
applications, and I think that if you look at every single 
one of those narrow applications there is a chance 
that it negatively affects women. I don’t think artificial 
intelligence is the issue here; it’s the additional issue 
rather than the cause. We’re talking about the risk 
that our unconscious sexism or unconscious racism 
seep into the machines that we’re building. 

How do we get anyone who’s building Al to think 
I about these things? We need to have consumers 
§ demand ethical Al. Not enough people are seeing 
g this as more than just a gender issue; this is an actual, 
I fundamental product issue, —as told to Rachel Metz 
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Constant Monitoring + Al = 
Rx for Personal Health 

An audacious Chinese entrepreneur wants to test your body for everything. 
But are computers really smart enough to make sense of all that data? 


I 

smart mirror isn’t very smart,” says Jun Wang, 
iing in front of a full-length mirror wearing designer 
ripped at the knees. “It’s just a camera and a mir- 
te says, looking mildly distressed—or as distressed 
as possible for a man whose face is perpetually unper¬ 
turbed. “What I want is a mirror that does a 3-D scan of 
me heije,” he says, using his hands to trace the contour 
of his thighs, “and here.” Wang indicates his belly, which 
is\,iean. v “We want an exact 3-D figure of you: the fat, the 
muscle—your entire body shape, plus facial recognition, 
and what’s going on with your skin.” He points to the top 
right area of the mirror. “And I want readouts about my 
health up there, next to where I’m brushing my teeth— 
my weight, blood pressure, and heart rate, and how does 
that correlate with my DNA?” 

This as-yet-unrealized smart mirror is just one of sev¬ 
eral gadgets that Wang, a 41-year-old biologist and com¬ 
puter scientist turned entrepreneur, says he is building. 

The devices will help gather, analyze, and display a crush 
of health data he wants to collect about himself—and, he 
hopes, from millions of others. This is why Wang cofounded 
iCarbonX (ICX), a highly ambitious, if quixotic, personal- 
health company based here in Shenzhen, in southern China. 


“Thi: 
stap 
jeans 
ror ” t 


ICX wants to capture more data about your body than 
has ever before been possible. It starts with your DNA 
sequence and includes data from Fitbit-style wearables 
that measure your steps, heart rate, and sleep patterns. 
Add frequent blood tests to measure various proteins and 
enzymes that can, say, reflect the health of your heart or 
signal very early signs of cancer. Include monitoring of 
the ever-changing levels of metabolites produced by the 
body as it processes food; traditional blood tests on lev¬ 
els of cholesterol and glucose; heart data from an EKG; 
and information from your medical history. The goal: 
continuous monitoring of your health and suggestions of 
adjustments you might make in your diet and behavior 
before you slip from being healthy into the early stages of 
an illness. 

This sounds a little like personalized medicine, which 
has been discussed for years. But for Wang, it’s not just 
about treating disease. It’s also about what might be 
termed personalized health. “Right now you don’t know 
about your temperature, or your pulse, or the microbes 
inside you that affect your emotions,” he says. “Or what 
to do if you have an allergy, or you want to lose weight 
because you’re fat.” 
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This vision of personal health monitoring is becom¬ 
ing achievable in part because of dramatic cost reductions 
for sequencing DNA and measuring the many thousands of 
biological compounds and processes that regulate the body. 
What all that means for any one of us, especially when all the 
readings are combined, is unclear. But ICX is part of a new 
wave of companies that figure they can find something mean¬ 
ingful in the data and enable medicine to stop merely react¬ 
ing to an illness you have; these companies want to keep you 
healthy at a fraction of the cost. Unlocking this puzzle, with 
its millions of moving pieces, is where AI and other advanced 
computing techniques will have to come in. “AI is how we can 
take all of this information and tell you things that you don’t 
know about your health,” says Wang. 

Assuming it works, putting all of this together will not 
be cheap. As CEO of ICX, Wang has raised $600 million in 
funding for the effort, a remarkable amount for a project 
offering high-tech tests for healthy people. “But he’ll need 
it, and probably more, with everything they want to test,” 
says Eric Schadt, a computational biologist and mathemati¬ 
cian who recently stepped down as director of Mount Sinai’s 
Icahn Institute for Genomics and Multiscale Biology in New 
York. Schadt has launched his own health data company, 
called Sema4, which is scanning genomes and molecular bio¬ 
markers. 

ICX is using its pile of cash in part to invest in or acquire 
companies that might contribute to Wang’s holistic vision. This 
includes a $161 million stake in Colorado-based SomaLogic, 
which is working on a chip that can measure 5,000 proteins in 


The devices will help 
gather, analyze, and display 
a crush of health data he 
wants to collect about 
himself-and, he hopes, 
from millions of others. 


the blood; more than $100 million in PatientsLikeMe, a com¬ 
pany in Cambridge, Massachusetts, that provides an online 
platform for more than 500,000 patients to share experiences, 
metrics, and feelings about their health and diseases; and $30 
million in AOBiome, also of Cambridge, which sells spray-on 
microbes that it says make skin healthier. ICX also recently 
invested in HealthTell of San Ramon, California, which iden¬ 
tifies antibodies from a blood sample as clues to the presence 
and progress of diseases including cancer and autoimmune 
disorders. Additionally, ICX is collaborating with several com¬ 
panies in China. 

Tying this eclectic alliance together is an aggressive effort 
to build an artificial-intelligence system that will attempt to 
analyze this tremendous amount of data. That’s being led by 
iCarbonX-Israel, which ICX acquired last year. Founded in 
2005 as Imagu Vision Technologies, the company develops 
software to interpret CT scans and other medical images. 
Now Imagu’s engineers are working with counterparts at ICX 
to create what they call a “virtual health brain” that will inter¬ 
pret the thousands of data points ICX wants to collect on 
each customer. “We want to create a tool that not only ana¬ 
lyzes data but offers ways to help people improve their health, 
like how to alter their diet,” says Imagu CEO and cofounder 
Mor Amitai. 

“If this all sounds ridiculously complicated, it is,” says 
Wang, smiling in a way that blends reassurance—which 
undoubtedly is appreciated by investors—and bemusement, 
as if he knows that what he is proposing sounds a bit daft. The 
question, then: can he use his money and technical savvy to 
revolutionize medicine? 

Precision health 

A tall man with short black hair, Wang strolls coolly through 
his company’s headquarters, a Silicon Valley knockoff with 
open workstations, glass-walled conference rooms, a gym, 
and a cafe always stocked with food, healthy drinks, tea, and 
coffee. It’s on the third floor of an industrial-park building 
in a complex of similarly unexceptional structures tucked 
between two sprawling, wooded theme parks called Happy 
Valley and the China Folk Culture Village. In the back of 
ICX’s HQ is Wang’s office, a comfortable niche with deep 
leather chairs and a private conference room—a business set¬ 
ting that is a long way from where Wang started, as an aca¬ 
demic researcher sequencing DNA at Beijing University in 
the late 1990s. 

Wang authored over 100 studies as a professor at the Uni¬ 
versity of Copenhagen and as a bioinformatics whiz at the Bei¬ 
jing Genomics Institute (BGI), which he helped found in 1999- 
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BGI was the organization that led China’s relatively small con¬ 
tribution to the Human Genome Project, a worldwide effort in 
which several countries worked on different segments of the 
human DNA sequence published in 2003. BGI later churned 
out the first complete DNA sequences of an Asian person, a 
strain of rice, the SARS virus, and the giant panda. During his 
stint as BGI’s CEO, Wang helped build the company into one 
of the largest sequencing operations in the world. In 2016, it 
had revenue of $250 million, and this summer it issued an 
IPO. Wang remains a major BGI shareholder and a member of 
the board. 

But he left the company in 2015 because he was frus¬ 
trated by the limits of genomics. Specifically, sequencing 
DNA doesn’t provide much insight into the health of most 
individuals. Scientists have found countless DNA markers 
that seem as if they should help determine whether a person 
is healthy or sick. But those markers have turned out, nearly 
15 years after the completion of the Human Genome Project, 
to make less of a difference than originally thought. With the 
exception of certain rare genetic mutations, DNA is just one 
determinant of a person’s medical fate. “It turns out you also 
need to know about proteins, and metabolites, and all the 
rest,” says Wang. 

Soon after his departure from BGI, Wang formed ICX, 
knowing he would do something with AI and health. But 
he wasn’t sure exactly what data besides DNA the company 
could, or should, collect. To figure it out, he met with a range 
of experts and companies, including a pivotal meeting in 
July 2016 at the Original Max’s restaurant in Burlingame, 
California, near the San Francisco airport. In town pitch¬ 
ing ICX to investors and prospective partners, Wang had 


“You also need millions of 
people - maybe as many as 
10 million people-to 
get meaningful signals for 
common diseases.” 
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arranged to see Jamie Heywood, the cofounder and chairman 
of PatientsLikeMe, who was visiting from Boston. As they 
sat in an orange-and-yellow plastic booth in the truck-stop- 
style cafe, it didn’t take Heywood and Wang long to realize 
that they shared a fundamental exasperation with the limita¬ 
tions of today’s medical practices. Giving people more data 
seemed like a promising route. PatientsLikeMe, which runs 
a service where thousands of members discuss their vari¬ 
ous chronic diseases in online forums and provide metrics 
about their health and the progression of their disease, had 
already shown the value of careful health tracking by indi¬ 
viduals. Drinking coffee, Wang and Heywood dissed clas¬ 
sic medical testing, which tends to be static, with one test 
taken at a time—an EKG in a clinical setting every year or 
two, for example, or when symptoms seem to warrant it. “We 
got excited about the possibility that we could discover the 
early stages of when a person shifts from good health to, say, 
becoming a diabetic,” says Heywood, an MIT-trained engi¬ 
neer. “We both agreed that the technology is there, or is close 
to being there.” 

Heywood, who is a fast-talking bundle of energy where 
Wang conveys a steady calm, suggested that such a profile 
should also include the sort of behavioral and personal data 
collected by his company. Information that people share in 
the forums of PatientsLikeMe—on such issues as the health 
impact of stress at work—provides valuable clues to other 
members on how they can better manage their chronic condi¬ 
tions. Why not help healthy people use similar tools and data? 
“It took about five minutes for Jun and I to realize that we 
could do this,” he recalls. 

Heywood brought something else to the table: his com¬ 
pany had built a computer platform designed to analyze data 
reported by its half-million users. But it’s not yet clear that 
combining all the data that ICX and its collaborators want to 
capture will be meaningful. Nor is it likely that AI will find 
significant correlations in the data unless ICX lures millions 
of people to its service—and even that many might not be 
enough. “ICX will struggle,” predicts Eric Schadt of Sema4. 
“You also need millions of people—maybe as many as 10 mil¬ 
lion people—to get meaningful signals for common diseases.” 

Wang readily acknowledges the challenges. “To do every¬ 
thing we want will take many years,” he says. When asked 
about the need to test large numbers of people to discern sig¬ 
nals in the noise of all this data, he says that ICX is looking to 
enroll at least one million people in the next five years. “China 
has this big population, so I’m not worried about this,” he says. 
He adds that as disposable income increases in China, people 
want to spend money on their health. 
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Wang admits, however, that he doesn’t yet have a clear 
business plan. “I tend to think about the right thing to do 
with the science and the product first,” he says. “Then I figure 
out the business model. Investors are okay with this. They 
don’t want short term.” 

Smoke and mirrors? 

Underlying ICX’s challenge are also some fundamental ques¬ 
tions about how to integrate artificial intelligence into health 
care. There’s little doubt that advanced computing will even¬ 
tually provide a huge boost toward making sense of all man¬ 
ner of health and biomedical data. And Wang is not the only 
one with business ambitions for the technology. According to 
CB Insights, which tracks venture capital investments, inves¬ 
tors are funding 106 startups in AI and health—up from a 
handful a few years ago. They’re pursuing everything from 
mental health and drug discovery to lifestyle management, 
virtual assistants, hospital management, and medical imag¬ 
ing and diagnostics. While this sounds impressive, AI so far 
has failed to make a substantial impact on most of medicine 
and health care. “In certain niches, AI is here and has been 
for years,” says Marty Kohn, a physician and the former chief 
medical scientist at IBM, who helped develop IBM Watson 
Health. “But it’s not happening at scale. And it hasn’t yet 
helped large numbers of patients.” 

One reason is that it’s incredibly difficult to interpret that 
much data. “I think AI has tremendous potential,” says Leroy 
Hood, the president and cofounder of the Institute for Sys- 
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terns Biology in Seattle. “But the claims for AI and health 
care are very overblown.” Most companies, he suggests, 

“don’t do real science.” 

A longtime pioneer in finding tools for understanding the 
body’s complex functions, Hood is a cofounder of Seattle-based 
Arivale, another health data company. In 2014, Arivale started 
offering its own version of lifestyle, wellness, and molecular 
testing, coupled with personal coaching. In July 2017, Hood 
and Arivale published a small study in Nature Biotechnology 
that he says provides a proof-of-concept analysis of what the 
researchers call “personal, dense, dynamic data clouds” mea¬ 
sured in healthy people over time. They used advanced algo¬ 
rithms to make correlations for 108 subjects who took dozens 
of health tests and measurements. Some of the participants 
learned that they had vitamin deficiencies; others found they 
had early signs of inflammatory bowel disease or diabetes that 
needed tending through diet or supplements. These results, 
however, are preliminary, and far more of them will be needed 
to separate real findings from the firehose of data. 

As for Wang, he is experimenting on himself with still more 
ways to acquire such information. As he continues his tour of 
ICX’s headquarters in Shenzhen, he points to a toilet just off 
his office where he collects plastic bags of poop for his daily 
microbiome analysis. Wang describes plans to build a “smart 
toilet” that will capture and analyze one’s waste and feed it into 
an Al-generated personal profile. “We have the technology 
to do this,” he says. “We have the algorithms. It will be cheap, 
something like $200.” Wang next lifts up his sky-blue polo 
shirt to show off a wireless continuous heart-rate monitor. 

One wonders, however, if millions of healthy people will 
be as obsessed as Jun Wang is with collecting so much data 
on themselves. The question seems to take him by surprise, 
momentarily roiling his composure. He knits his brow, looking 
as if he can’t imagine that other people might not want smart 
mirrors and toilets, frequent blood draws to measure thou¬ 
sands of metabolites, and heart monitors taped to their chests. 
“I’m not asking everyone to do this,” he finally says. “People 
choose not to know a lot of things. But there are plenty of peo¬ 
ple who want to know, or can be educated to want to know.” 

He pauses for another nanosecond and then flashes that smile, 
looking as if he had just figured out the answer to this literally 
multibillion-dollar question about his effort and his company. 
“People used to not want to know their genes; now more and 
more people want to know,” he says. “I’m sure that this trend 
will continue.” E3 


David Ewing Duncan is a life science journalist and author of 
Experimental Man, and the CEO of Arc Fusion. 
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How to Root Out 
Hidden Biases in Al 

Algorithms are making life-changing decisions like 
denying parole or granting loans. Cynthia Dwork, a 
computer scientist at Harvard, is developing ways of 
making sure the machines are operating fairly. 


Why is it hard for algorithm designers or 
data scientists to account for bias and 
unfairness? 

Take a work environment that’s hos¬ 
tile to women. Suppose that you define 
success as the fact that someone holds 
a job for two to three years and gets a 
promotion. Then your predictor—based 
on the historical data—will accurately pre¬ 
dict that it’s not a good idea to hire women. 
What’s interesting here is that we’re not 
talking about historical hiring decisions. 
Even if the hiring decisions were totally 
unbiased, the reality—the real discrimina¬ 
tion in the hostile environment—persists. 
It’s deeper, more structural, more ingrained 
and harder to overcome. 

I believe the great use for machine 
learning and Al will be in conjunction with 
really knowledgeable people who know 
history and sociology and psychology to 
figure out who should be treated similarly 
to whom. 

I’m not saying computers will never 
be able to do it, but I don’t see it now. 
How do you know when you have the 
right model, and when it’s capturing what 
really happened in society? You need to 
I have an understanding of what you’re talk- 
§ ing about. There’s the famous saying “All 
g models are wrong and some are useful.” 
i —as told to Will Knight 
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Inside the Moonshot Effort to Finally 




CHTMAN LAB (CUBE) 


Al is only loosely modeled on the 
brain. So what if you wanted to 
do it right? Youd need to do what 
has been impossible until now: 
map what actually happens in 
neurons and nerve fibers. 
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Previous pages: A rat brain in a dish and a rendering of two neurons with 
spiny dendrites. This page, top: A technician observes the brain of alive rat 
during a test. Bottom: After the test, the animal’s brain has been removed. 
Opposite page: The brain is glued to a plate before being scanned. 


says David Cox. Yes, it has gotten astonish¬ 
ingly good, from near-perfect facial recogni¬ 
tion to driverless cars and world-champion 
Go-playing machines. And it’s true that some 
AI applications don’t even have to be pro¬ 
grammed anymore: they’re based on architec¬ 
tures that allow them to learn from 
experience. 

Yet there is still something clumsy and 
brute-force about it, says Cox, a neurosci¬ 
entist at Harvard. “To build a dog detector, 
you need to show the program thousands 
of things that are dogs and thousands that 
aren’t dogs,” he says. “My daughter only had 
to see one dog”—and has happily pointed out 
puppies ever since. And the knowledge that 
today’s AI does manage to extract from all 
that data can be oddly fragile. Add some art¬ 
ful static to an image—noise that a human 
wouldn’t even notice—and the computer 
might just mistake a dog for a dumpster. 
That’s not good if people are using facial rec¬ 
ognition for, say, security on smartphones. (See “Is AI Riding a 
One-Trick Pony?” on page 28.) 

To overcome such limitations, Cox and dozens of other 
neuroscientists and machine-learning experts joined forces 
last year for the Machine Intelligence from Cortical Net¬ 
works (MICrONS) initiative: a $100 million effort to reverse- 
engineer the brain. It will be the neuroscience equivalent of a 
moonshot, says Jacob Vogelstein, who conceived and launched 
MICrONS when he was a program officer for the Intelligence 
Advanced Research Projects Agency, the U.S. intelligence com¬ 
munity’s research arm. (He is now at the venture capital firm 
Camden Partners in Baltimore.) MICrONS researchers are 
attempting to chart the function and structure of every detail 
in a small piece of rodent cortex. 
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It’s a testament to the brain’s complexity that a moonshot is 
needed to map even this tiny piece of cortex, a cube measuring 
one millimeter on a side—the size of a coarse grain of sand. But 
this cube is thousands of times bigger than any chunk of brain 
anyone has tried to detail. It will contain roughly 100,000 neu¬ 
rons and something like a billion synapses, the junctions that 
allow nerve impulses to leap from one neuron to the next. 

It’s an ambition that leaves other neuroscientists awe¬ 
struck. “I think what they are doing is heroic,” says Eve Marder, 
who has spent her entire career studying much smaller neural 
circuits at Brandeis University. “It’s among the most exciting 
things happening in neuroscience,” says Konrad Kording, who 
does computational modeling of the brain at the University of 
Pennsylvania. 

The ultimate payoff will be the neural secrets mined from 
the project’s data—principles that should form what Vogelstein 
calls “the computational building blocks for the next generation 
of AI.” After all, he says, today’s neural networks are based on a 
decades-old architecture and a fairly simplistic notion of how 
the brain works. Essentially, these systems spread knowledge 
across thousands of densely interconnected “nodes,” analogous 
to the brain’s neurons. The systems improve their performance 
by adjusting the strength of the connections. But in most com¬ 
puter neural networks the signals always cascade forward, from 
one set of nodes to the next. The real brain is full of feedback: 
for every bundle of nerve fibers conveying signals from one 
region to the next, there is an equal or greater number of fibers 
coming back the other way. But why? Are those feedback fibers 
the secret to one-shot learning and so many other aspects of the 
brain’s immense power? Is something else going on? 

MICrONS should provide at least some of the answers, says 
Princeton University neuroscientist Sebastian Seung, who is 
playing a key role in the mapping effort. In fact, he says, “I don’t 
think we can answer these questions without a project like this.” 

The MICrONS teams—one led by Cox, one based at Rice 
University and the Baylor College of Medicine, and a 
third at Carnegie Mellon—are each pursuing something 
that is remarkably comprehensive: a reconstruction of all 
the cells in a cubic millimeter of a rat’s brain, plus a wir¬ 
ing diagram—a “connectome”—showing how every cell 
is connected to every other cell, and data showing exactly 
which situations make neurons fire and influence other neurons. 

The first step is to look into the rats’ brains and figure out 
what neurons in that cubic millimeter are actually doing. When 
the animal is given a specific visual stimulus, such as a line ori¬ 
ented a certain way, which neurons suddenly start firing off 
impulses, and which neighbors respond? 

As recently as a decade ago, capturing that kind of data 
ranged from difficult to impossible: “The tools just never 
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Top: The small cube in the upper left is the portion of the brain that will 
be mapped. Bottom: That piece of the brain is encased in acrylic in 
preparation for being sliced extremely thin. 
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existed,” Vogelstein says. It’s true that researchers could slide 
ultrathin wires into the brain and get beautiful recordings 
of electrical activity in individual neurons. But they couldn’t 
record from more than a few dozen at a time because the cells 
are packed so tightly together. Researchers could also map 
the overall geography of neural activity by putting humans 
and other animals in MRI machines. But researchers couldn’t 
monitor individual neurons that way: the spatial resolution 
was about a millimeter at best. 

What broke that impasse was the development of techniques 
for making neurons light up when they fire in a living brain. To 
do it, scientists typically seed the neurons with fluorescent pro¬ 
teins that glow in the presence of calcium ions, which surge in 
abundance whenever a cell fires. The proteins can be inserted 
into a rodent’s brain chemically, carried in by a benign virus, or 
even encoded into the neurons’ genome. The fluorescence can 
then be triggered in several ways—perhaps most usefully, by a 
pair of lasers that pump infrared light into the rat through a win¬ 
dow set into its skull. The infrared frequencies allow the photons 
to penetrate the comparatively opaque neural tissue without 
damaging anything, before getting absorbed by the fluorescent 
proteins. The proteins, in turn, combine the energy from two of 
the infrared photons and release it as a single visible-light pho¬ 
ton that can be seen under an ordinary microscope as the animal 
looks at something or performs any number of other actions. 

Andreas Tolias, who leads part of the team at Baylor, says 
this is “revolutionary” because “you can record from every sin¬ 
gle neuron, even those that are right next to one another.” 

Once a team in Cox’s lab has mapped a rat’s neural activ¬ 
ity, the animal is killed and its brain is infused with the heavy 
metal osmium. Then a team headed by Harvard biologist Jeff 
Lichtman cuts the brain into slices and figures out exactly how 
the neurons are organized and connected. 

That process starts in a basement lab with a desktop 
machine that works like a delicatessen salami slicer. A small 
metal plate rises and falls, methodically carving away the tip of 
what seems to be an amber-colored crayon and adhering the 
slices to a conveyor belt made of plastic tape. The difference is 
that the “salami” is actually a tube of hard resin that encases 
and supports the fragile brain tissue, the moving plate contains 
an impossibly sharp diamond blade, and the slices are about 
30 nanometers thick. 

Next, at another lab down the hall, lengths of tape contain¬ 
ing several brain slices each are mounted on silicon wafers and 
placed inside what looks like a large industrial refrigerator. The 
device is an electron microscope: it uses 61 electron beams to 
scan 61 patches of brain tissue simultaneously at a resolution of 
four nanometers. 

Each wafer takes about 26 hours to scan. Monitors next 
to the microscope show the resulting images as they build 



Top: The cut slices of brain stick to a plastic tape. Bottom: The tape, 
with brain samples attached, is trimmed and put on a slide plate that will 
go into a huge scanning machine. 
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up in awe-inspiring detail—cell membranes, mitochondria, 
neurotransmitter-filled vesicles crowding at the synapses. It’s 
like zooming in on a fractal: the closer you look, the more com¬ 
plexity you see. 

Slicing is hardly the end of the story. Even as the scans come 
pouring out of the microscope—“You’re sort of making a movie 
where each slice is deeper,” says Lichtman—they are forwarded 
to a team led by Harvard computer scientist Hanspeter Pfister. 
“Our role is to take the images and extract as much information 
as we can,” says Pfister. 

That means reconstructing all those three-dimensional 
neurons—with all their organelles, synapses, and other fea¬ 
tures—from a stack of 2-D slices. Humans could do it with 
paper and pencil, but that would be hopelessly slow, says Pfis¬ 
ter. So he and his team have trained neural networks to track 
the real neurons. “They perform a lot better than all the other 
methods we’ve used,” he says. 

Each neuron, no matter its size, puts out a forest of ten¬ 
drils known as dendrites, and each has another long, thin fiber 
called an axon for transmitting nerve impulses over long dis¬ 
tances—completely across the brain, in extreme cases, or even 
all the way down the spinal cord. But by mapping a cubic mil¬ 
limeter as MICrONS is doing, researchers can follow most of 
these fibers from beginning to end and thus see a complete 
neural circuit. “I think we’ll discover things,” Pfister says. 
“Probably structures we never suspected, and completely new 
insights into the wiring.” 

Among the questions the MICrONS teams hope to 
begin answering: What are the brain’s algorithms? 
How do all those neural circuits actually work? And 
in particular, what is all that feedback doing? 

Many of today’s AI applications don’t use feed¬ 
back. Electronic signals in most neural networks 
cascade from one layer of nodes to the next, but 
generally not backward. (Don’t be thrown by the 
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term “backpropagation,” which is a way to train neural net¬ 
works.) That’s not a hard-and-fast rule: “recurrent” neural 
networks do have connections that go backward, which helps 
them deal with inputs that change with time. But none of 
them use feedback on anything like the brain’s scale. In one 
well-studied part of the visual cortex, says Tai Sing Lee at 
Carnegie Mellon, “only 5 to 10 percent of the synapses are lis¬ 
tening to input from the eyes.” The rest are listening to feed¬ 
back from higher levels in the brain. 

There are two broad theories about what the feedback is for, 
says Cox, and “one is the notion that the brain is constantly try¬ 
ing to predict its own inputs.” While the sensory cortex is pro¬ 
cessing this frame of the movie, so to speak, the higher levels of 
the brain are trying to anticipate the next frame, and passing 
their best guesses back down through the feedback fibers. 

This the only way the brain can deal with a fast-moving 
environment. “Neurons are really slow,” Cox says. “It can take 
up to 170 to 200 milliseconds to go from light hitting the ret¬ 
ina through all the stages of processing up to the level of con¬ 
scious perception. In that time, Serena Williams’s tennis serve 
travels nine meters.” So anyone who manages to return that 
serve must be swinging her racket on the basis of prediction. 

And if you’re constantly trying to predict the future, Cox 
says, “then when the real future arrives, you can adjust to 
make your next prediction better.” That meshes well with the 
second major theory being explored: that the brain’s feed¬ 
back connections are there to guide learning. Indeed, com¬ 
puter simulations show that a struggle for improvement forces 
any system to build better and better models of the world. 

For example, Cox says, “you have to figure out how a face will 
appear if it turns.” And that, he says, may turn out to be a criti¬ 
cal piece of the one-shot-learning puzzle. 

“When my daughter first saw a dog,” says Cox, “she didn’t 
have to learn about how shadows work, or how light bounces 
off surfaces.” She had already built up a rich reservoir of expe¬ 
rience about such things, just from living in the world. “So 


Opposite page, top row: Scans of brain slices are stitched together 
by an algorithm. Middle row: A “multibeam field of view,” made of 61 
images taken by the electron microscope, is seen at left; 14 multibeam 
fields of view are combined at right. Bottom row: Scans are assembled 
into a cube and colorized. 
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when she got to something like ‘That’s a dog,”’ he says, “she 
could add that information to a huge body of knowledge.” 

If these ideas about the brain’s feedback are correct, they 
could show up in MICrONS’s detailed map of a brain’s form 
and function. The map could demonstrate what tricks the neu¬ 
ral circuitry uses to implement prediction and learning. Even¬ 
tually, new AI applications could mimic that process. 

Even then, however, we will remain far from answering all 
the questions about the brain. Knowing neural circuitry won’t 
teach us everything. There are forms of cell-to-cell commu¬ 
nication that don’t go through the synapses, including some 
performed by hormones and neurotransmitters floating in the 
spaces between the neurons. There is also the issue of scale. As 
big a leap as MICrONS may be, it is still just looking at a tiny 
piece of cortex for clues about what’s relevant to computation. 
And the cortex is just the thin outer layer of the brain. Critical 
command-and-control functions are also carried out by deep- 
brain structures such as the thalamus and the basal ganglia. 

The good news is that MICrONS is already paving the way 
for future projects that map larger sections of the brain. 

Much of the $100 million, Vogelstein says, is being spent 
on data collection technologies that won’t have to be invented 
again. At the same time, MICrONS teams are developing 
faster scanning techniques, including one that eliminates the 
need to slice tissue. The Carnegie Mellon group, working with 
teams at Harvard, MIT, and the Woods Hole Oceanographic 
Institution, has devised a way to uniquely label each neuron 
with a “bar-coding” scheme and then view the cells in great 
detail by saturating them with a special gel that very gently 
inflates them to dozens or hundreds of times their normal size. 

“So the first cubic millimeter will be hard to collect,” 
Vogelstein says, “but the next will be much easier.” H 


M. Mitchell Waldrop is a freelance writer in Washington, D.C. He 
is the author of Complexity and The Dream Machine and was 
formerly an editor at Nature. 



On these pages: The colorized cubes are useful in 3-D illustrations of 
various neuronal structures and processes, giving scientists their most 
detailed map yet of what actually happens in the brain. 
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Fiction That Gets Al Right 


Even the most futuristic applications of Al, 
from robotic servants to instant health scans, 
somehow already seem familiar because 
they have been endless fodder for pop 
culture. At the movies, you’ve probably seen 
artificial intelligence as a menace in The 
Terminator or 2001: A Space Odyssey; as a 
near-destroyer of the world in WarGames; as 
our master in The Matrix; or as a companion 
(as in Her and in the Stanley Kubrick-Steven 
Spielberg combination that was called A.I. 
Artificial Intelligence). But many lesser-known 
stories, plays, TV shows, and films stand out 
as well for their perceptiveness about how 
sophisticated Al will affect us. Here are a few 
suggestions. —Brian Bergstein 
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The Machine Stops, by e.m. forster, 1909 

This short story takes place in a future chillingly akin to the Singularity as 
envisioned by Ray Kurzweil, except here it’s the final step before the end 
of the world rather than the epitome of computer-assisted enlightenment. 
Everyone sits alone in an underground room, listening to lectures from, or 
giving lectures to, friends who are always remote; they appear to each 
other in something like holograms, because life is now only about the 
endless discussion of ideas rather than physical experience. The Machine 
takes care of all their other needs—for a time, at least. 


Humanity, in its desire for 
comfort, had over-reached 
itself. It had exploited the 
riches of nature too far. 
Quietly and complacently, it 
was sinking into decadence, 
and progress had come to 
mean the progress of the 
Machine. 



R.U.R. (Rossum’s Universal Robots), by karel capek, 1920 
This play introduced “robot” into the language, from the Czech word 
“robota,” which means “forced labor.” It’s all about the hubris of a factory 
owner unable to resist the temptation to automate as much labor as 
possible—until the “artificial people” he creates turn on him and the rest 
of humanity. 


‘“Robots of the world! We, the 
first union at Rossum’s 
Universal Robots, declare 
that man is our enemy and 
the blight of the universe.’ 
Who the hell taught them to 
use phrases like that?” 
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There Will Come Soft Rains, by ray bradbury, 1950 ‘‘Mrs. McClellan, which poem 

A smart home in post-apocalypse 2026 has outlived its human occupants would you like this evening?” 

but still cooks for them and tries to guide them through their daily The house was silent The 

routines. voice said at last, ‘‘Since you 

express no preference, I shall 
select a poem at random.” 



Humans, debuted in 2015 

Imagine the world of today, except that artificial people called Synths 
handle the drudgery in factories, farm fields, and homes. This TV series is 
gripping because it explores the implications on many levels. How does 
robotic labor erode humans’ motivation and sense of purpose? And what 
if some of these Synths can feel and think for themselves? There’s a whiff 
of R.U.R. to it, and in fact a character who illegally hacks robots has the 
last name of Capek. 


Hobb: ‘‘Robert, these 
machines are conscious.” 

Robert: ‘‘How do you know 
they don’t just simulate it?” 

Hobb: ‘‘How do we know you 
don’t?” 



Chappie, 2015 

Unlike a machine encoded with the Three Laws of Robotics in Isaac 
Asimov’s stories, the police droid named Chappie in this movie doesn’t 
know from the start how to behave. He has to learn right and wrong by 
observing the world. That childlike innocence becomes problematic when 
he gets stolen by two gangsters who try to use him for crime. As Gary 
Marcus, a psychologist and Al entrepreneur, wrote in the New Yorker, 
‘‘Chappie can be seen an impassioned plea for moral education, not just 
for humans but for our future silicon-based companions.” 


‘‘Chappie, in life, lots of 
people will tell you what you 
can’t do. And you know what, 
you must never listen.” 



Speak, by louisa hall, 2015 

The recollections of an Al doll creator and four other characters—who all 
live at different points in time from 1663 to 2040—are interwoven in this 
novel to explore fundamental similarities between human and computer 
intelligence, memory, and expression. (For more by Hall, see “How We 
Feel About Robots That Feel,” page 74.) 


Ramona learned for the sake 
of her doll. She ran with her 
doll so her doll could feel 
movement. The two of them 
never fought. They were 
perfect for each other. My 
daughter’s doll was a softly 
blurred mirror that I held up to 
her face. 
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On a tropical island that marks the southern tip of China, a 
computer program called Lengpudashi is playing one-on- 
one poker against a dozen people at once, and it’s absolutely 
crushing them. Lengpudashi, which means “cold poker mas¬ 
ter” in Mandarin, is using a new artificial-intelligence tech¬ 
nique to outbet and outbluff its opponents in a two-player 
version of Texas hold ’em. 

The venue for the tournament is a modern-looking tech¬ 
nology park in Haikou, capital of the island of Hainan. Out¬ 
side, modern high-rises loom over aging neighborhoods. 

Those gathered to play the machine include several poker 
champs, some well-known Chinese investors, entrepreneurs, 
and CEOs, and even the odd television celebrity. The games 
are being broadcast online, and millions are watching. The 
event symbolizes a growing sense of excitement and enthu¬ 
siasm for artificial intelligence in China, but there’s also a 
problem. Lengpudashi wasn’t made in Hainan, Beijing, or 
Shanghai; it was built in Pittsburgh, U.S.A. 

For many in China, this simply won’t do. The country is 
now embarking on an unprecedented effort to master artifi¬ 
cial intelligence. Its government is planning to pour hundreds 
of billions of yuan (tens of billions of dollars) into the tech¬ 
nology in coming years, and companies are investing heavily 
in nurturing and developing AI talent. If this country-wide 
effort succeeds—and there are many signs it will—China 
could emerge as a leading force in AI, improving the produc¬ 
tivity of its industries and helping it become leader in creating 
new businesses that leverage the technology. And if, as many 
believe, AI is the key to future growth, China’s prowess in the 
field will help fortify its position as the dominant economic 
power in the world. 

Indeed, the country’s political and business leaders 
are betting that AI can jump-start its economy. In recent 
decades, a booming manufacturing sector—and market 
reforms encouraging foreign trade and investment—have 
helped bring hundreds of millions of people out of poverty, 
creating business empires and transforming Chinese soci¬ 
ety. But manufacturing growth is slowing, and the country 
is looking toward a future built around advanced technol¬ 
ogy (see “China Is Building a Robot Army of Model Workers,” 
May/June 2016). Applying artificial intelligence may be the 
next step in this technology-fueled economic miracle. While 
many in the West fret about AI eliminating jobs and worsen¬ 
ing wealth and income inequality, China seems to believe it 
can bring about precisely the opposite outcomes. 

China’s AI push includes an extraordinary commitment 
from the government, which recently announced a sweeping 
vision for AI ascendancy. The plan calls for homegrown AI to 


match that developed in the West within three years, for Chi¬ 
na’s researchers to be making “major breakthroughs” by 2025, 
and for Chinese AI to be the envy of the world by 2030. 

There are good reasons to believe the country can make 
this vision real. In the early 2000s, the government said 
it wanted to build a high-speed rail network that would 
spur technological development and improve the country’s 
transportation system. This train network is now the most 
advanced in the world. “When the Chinese government 
announces a plan like this, it has significant implications for 
the country and the economy,” says Andrew Ng, a prominent 
AI expert who previously oversaw AI technology and strat¬ 
egy at China’s biggest online search company, Baidu. “It’s a 
very strong signal to everyone that things will happen.” 

The government’s call to action will accelerate what has 
already begun to happen. The country’s tech companies, led 
by the Internet giants Baidu, Alibaba, and Tencent, are hir¬ 
ing scores of AI experts, building new research centers, and 
investing in data centers that rival anything operated by Ama¬ 
zon, Google, or Microsoft. Money is also pouring into count¬ 
less startups as Chinese entrepreneurs and investors spy a huge 
opportunity to harness AI in different industries. 

China has some big advantages in AI. It has a wealth of 
talented engineers and scientists, for one. It also is rich in the 
data necessary to train AI systems. With fewer obstacles to 
data collection and use, China is amassing huge databases 
that don’t exist in other countries. The results can be seen in 
the growth of facial-recognition systems based on machine 
learning: they now identify workers at offices and customers 
in stores, and they authenticate users of mobile apps. 

The nationwide interest in the poker tournament in 
Hainan reflects China’s appetite for the latest artificial- 
intelligence breakthroughs. Mastering even a two-player form 
of poker is a significant achievement for AI because, unlike 
many other games, poker requires players to act with limited 
information, and to sow uncertainty by behaving unpredict¬ 
able An optimal strategy therefore requires both careful and 
instinctive judgment, which are not easy qualities to give a 
machine. Lengpudashi impressively solved the problem by 
using a brilliant new game-theory algorithm, which could be 
very useful in many other scenarios, including financial trad¬ 
ing and business negotiations. But Lengpudashi has received 
far less attention in its home country than it has in Hainan. 

To explore China’s AI revolution and its implications, I’ve 
traveled to the heart of this boom and met with many of the 
key researchers, entrepreneurs, and executives. From China’s 
bustling capital to its factory-filled south, and from an ambi¬ 
tious new research center to a billion-dollar startup, one 
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Kai-Fu Lee is CEO of the venture capital firm Sinovation Ventures. 


thing is clear: artificial intelligence may have been invented 
in the West, but you can see its future taking shape on the 
other side of the world. 



LOOK EAST 


My journey begins at MIT, one of the wellsprings of artificial 
intelligence. Kai-Fu Lee, a well-known Chinese AI expert and 
investor and one of the organizers of the Hainan tournament, 
has come to recruit students for a new AI institute that his 
company, Sinovation Ventures, is building in Beijing. 

Lee gives a talk entirely in Mandarin to an auditorium 
packed with about 300 Chinese students. He is dressed 
impeccably, in an expensive-looking suit and dress shirt, 
and he speaks in a confident, soothing tone. The talk touches 
on the interwoven trends that have driven the recent rise in 
machine intelligence: more powerful computers, ingenious 
new algorithms, and huge quantities of data. He argues that 
China is perfectly poised to take advantage of these advances. 

“The U.S. and Canada have the best AI researchers in the 
world, but China has hundreds of people who are good, and 
way more data,” he tells the audience. “AI is an area where you 
need to evolve the algorithm and the data together; a large 
amount of data makes a large amount of difference.” 


In 1998 Lee founded Microsoft’s Beijing research lab, which 
showcased the country’s exciting talent pool (see “An Age of 
Ambition,” June 2004). Then, in 2005, he became the found¬ 
ing president of Google China. Lee is now famous for mentor¬ 
ing young entrepreneurs, and he has more than 50 million 
followers on the Chinese microblogging platform Sina Weibo. 

In the audience are exactly the type of prized students 
who would normally flock to Silicon Valley. But many are 
clearly taken by Lee’s message of opportunities in China. 

The crowd hangs on his every word, and some people clamor 
for autographs afterward. “Today the U.S. has a technology 
leadership,” Lee tells me later. “But China has a tremendous 
amount of potential.” 

To see what this potential looks like up close, I travel to 
Lee’s new institute, half a world away from MIT, in Beijing’s 
Haidian district. The streets outside are filled with people on 
colorful ride-sharing bikes. I pass lots of fashionable-looking 
young techies as well as people delivering breakfast—ordered 
via smartphone, no doubt—to busy workers. At the time of 
my visit, a major AI event is taking place a few hundred miles 
to the south in Wuzhen, a picturesque town of waterways. 
AlphaGo, a program developed by researchers at the Alpha¬ 
bet subsidiary DeepMind, is playing the ancient board game 
Go against several top Chinese players, including the world’s 
number one, Ke Jie. And it’s soundly beating them. 

AlphaGo’s victories in Wuzhen are followed closely in 
the Chinese capital. As I enter Sinovation’s institute, in fact, 

I notice a Go board on which engineers are testing out the 
moves made during some of the matches. 

The location of the institute is well chosen. From the 
office windows, you can see the campuses of both Peking 
University and Tsinghua University, two of China’s top aca¬ 
demic institutions. Sinovation provides machine-learning 
tools and data sets to train Chinese engineers, and it offers 
expertise for companies hoping to make use of AI. The insti¬ 
tute has about 30 full-time employees so far, but the plan is 
to employ more than 100 by next year, and to train hundreds 
of AI experts each year through internships and boot camps. 
Right now, roughly 80 percent of the institute’s funding and 
projects are aimed at commercializing AI, while the rest is 
focused on more far-out technology research and startup 
incubation. 

The goal isn’t to invent the next AlphaGo, though; it’s to 
upgrade thousands of companies across China using AI. Lee 
says many Chinese businesses, including the big state-owned 
enterprises, are technologically backward and ripe for over¬ 
haul, but they lack any AI expertise themselves. Needless to 
say, this presents an enormous opportunity. 
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Al EVERYWHERE 

Across the capital, in fact, I notice a remarkable amount 
of interest in artificial intelligence. In one restaurant, for 
instance, I find a machine that takes my picture and then sup¬ 
posedly uses Al to determine how healthy I am. This seems 
completely impossible, but the machine says I’m in great 
shape before suggesting that I have plenty to eat. 

This fascination with the technology is reflected in Bei¬ 
jing’s feverish startup scene, which is already producing some 
formidable Al companies. One of them is SenseTime, which 
was founded in 2014 and is already among the world’s most 


Earlier this year SenseTime’s engineers developed a novel 
image-processing technique for automatically removing 
smog and rain from photographs, and another for tracking 
full-body motion using a single camera. Last year they were 
part of a team that won a prestigious international computer- 
vision award. 

Xiaoou Tang, SenseTime’s founder and a professor at the 
Chinese University of Hong Kong, is wearing a suede jacket, 
slacks, and glasses, and he has an intense air about him. He 
seems fiercely proud of his company’s achievements. Tang 
explains that the company’s name comes from a phonetic 
transcription of the name of the Shang dynasty and of its first 



“China was leading the world then. 


valuable Al startups. Launched by researchers from the Chi¬ 
nese University of Hong Kong, SenseTime provides computer- 
vision technology to big Chinese companies, including the 
state-owned cellular provider China Mobile and the online 
retail giant JD.com. The company is now studying markets 
such as automotive systems. This July, SenseTime raised $410 
million in funding, giving it a valuation of $1.5 billion. The 
entrance to SenseTime’s office features several large screens 
fitted with cameras. One can automatically add augmented- 
reality effects to a person’s face. Snapchat and Instagram 
offer similar gimmicks, but this one can also add effects in 
response to hand and body movements as well as smiles or 
winks. 

Qing Luan, director of SenseTime’s augmented-reality 
group, previously developed office apps for Microsoft in Red¬ 
mond, Washington. She says she returned to China because 
the opportunities just seemed much bigger. “We were strug¬ 
gling to get a thousand users; then I talked with my friend who 
was working at a startup in China, and she said, ‘Oh, a million 
users is nothing—we get that in several days,”’ she recalls. 


ruler, Tang. This era, beginning around 1600 bce, was a criti¬ 
cal age of development for the country. “China was leading 
the world then,” Tang says with a smile. “And in the future, we 
will lead again with technological innovations.” 

MANUFACTURING INTELLIGENCE 

In the United States and other Western nations, many large 
sectors, such as manufacturing and services, have been slow 
to invest in Al and change their business practices. In China, 
there appears to be a greater sense of urgency about adapting 
to the changing technology. Across just about every industry, 
Chinese companies are shrugging off their reputation for fol¬ 
lowing Western businesses, and investing heavily in research 
and development. Ng, who previously led Baidu’s Al effort, 
says China’s business leaders understand better than most 
the need to embrace new trends. “The titans of industry [in 
China] have seen fortunes made and fortunes lost all within 
their lifetime,” he says. “When you see the tech trends shift, 
you had better move quickly, or someone else will beat you.” 
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Baidu anticipated the potential of artificial intelligence and 
sought to leverage it to reinvent its whole business. In 2014, 
the company created a lab dedicated to applying deep learn¬ 
ing across its business, and in recent years, its researchers 
have made some significant advances. When Microsoft devel¬ 
oped a system capable of better-than-human performance in 
speech recognition last year, for instance, few Western report¬ 
ers realized that Baidu had done that a year earlier. 

Following Baidu’s example, other Chinese tech companies 
are also looking to reinvent themselves with AI. The Inter¬ 
net leader Tencent, headquartered in the city of Shenzhen, is 
among them. 


already found in the U.S. This would evolve into WeChat, an 
innovative mobile platform that now supports social network¬ 
ing, news, games, and mobile payments. With 889 million 
daily active users, WeChat now has an incredible grip on Chi¬ 
na’s Internet market. 

Although Tencent created an AI lab only last year, it has 
hired scores of researchers and opened an outpost in Seattle. 
The company’s researchers have already copied AI innova¬ 
tions from the West, including DeepMind’s AlphaGo. Ten¬ 
cent’s AI lab is led by Tong Zhang, a quiet man with thin 
glasses and a round face, who previously worked at Baidu’s 
AI lab and before that was a professor at Rutgers Univer- 





And in the future, we will lead again.” 


Shenzhen is nestled next to Hong Kong in southern China. 
Approaching by air, I see an armada of cargo ships moored 
in the South China Sea. In 1980, when Shenzhen was a small 
market town, it was designated China’s first Special Economic 
Zone, granting it unprecedented economic and regulatory 
freedoms. Manufacturing empires were built on the backs of 
migrant workers producing every imaginable product, and 
the population rose from 30,000 to more than 11 million. 

In recent years, the city has reflected China’s technological 
progress, and it is now home to global technology compa¬ 
nies including the networking giant Huawei, the smartphone 
maker ZTE, and the electric-car company BYD. 

The city’s main strip is lined with palm trees, gaudy 
hotels, and busy bars and restaurants. Tencent’s headquar¬ 
ters, in Nanshan district, spans several large buildings, and 
the entrance is as busy as a subway station. Stepping inside, 
out of stifling humidity, I begin a tour that touts Tencent’s his¬ 
tory and achievements. And it shows that you don’t have to be 
first in a technology to have a big impact. In 2011, the com¬ 
pany launched a simple messaging app, modeled on products 


sity. Zhang speaks quietly, usually after a careful pause. He 
explains that AI will be crucial to Tencent’s plans to grow, 
especially outside China. “AI is important for the next phase,” 
he says. “At a certain stage, you just cannot copy things. You 
need your own capabilities.” 

I ask him if Tencent might be planning some spectacular 
demonstrations of AI, something like AlphaGo or Lengpu- 
dashi. Tencent owns several very popular games, including 
the strategy title League of Legends, which is played by more 
than 100 million people every month. Like Go, it requires 
instinctive actions, and like poker, it involves playing with¬ 
out a clear picture of your opponents’ standing. But it also 
requires planning far ahead, so it would be a worthy game for 
AI researchers to tackle next. “Right now, we have a bunch 
of small projects—some are more adventurous,” is all Zhang 
will say. 

Tencent’s AI goals may in fact be more practical. The com¬ 
pany has an amazing amount of conversation data thanks to 
WeChat and another messaging platform, called QQ. This 
data might be used to train machine-learning systems to hold 
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The headquarters of the Chinese e-commerce giant Alibaba is in Hangzhou. 


more meaningful conversations. Making advances in lan¬ 
guage could have countless practical applications, from bet¬ 
ter document analysis and search to much smarter personal 
assistants. “The challenge, and also the opportunity, will be in 
natural language,” Zhang says. 


/AlTmp 
THINKBIG 


It might be unnerving for Western nations to see a newcomer 
mastering an important technology, especially when the full 
potential of that technology remains uncertain. But it is wrong 
to view this story simply in terms of competition with the West. 

A big problem facing both the U.S. and China is slowing eco¬ 
nomic growth. While AI may eliminate certain jobs, it also has 
the potential to greatly expand the economy and create wealth 
by making many industries far more efficient and productive. 
China has embraced that simple fact more eagerly and more 
completely than many Western nations. But there’s no reason 
why China’s Al-fueled economic progress should come at the 
expense of other countries, if those countries embrace the same 
technology just as keenly. 


China might have unparalleled resources and enor¬ 
mous untapped potential, but the West has world-leading 
expertise and a strong research culture. Rather than 
worry about China’s progress, it would be wise for Western 
nations to focus on their existing strengths, investing heav¬ 
ily in research and education. The risk is missing out on an 
incredibly important technological shift. Yes, companies 
like Google and Facebook are making important strides in 
AI today, but this isn’t enough to reboot a whole economy. 
Despite the fanfare around AI, there are few economic 
signs—such as increased productivity—that most of the 
economy is taking advantage of the technology yet. Large 
segments of the economy beyond Silicon Valley, like medi¬ 
cine, service industries, and manufacturing, also need to 
sign on. 

I can’t help thinking of the poker tournament in Hainan 
and reflecting that the rest of the world should take inspira¬ 
tion from Lengpudashi, the poker-playing AI. It’s time to fol¬ 
low China’s lead and go all in on artificial intelligence. 


Will Knight is MIT Technology Review ’s senior editor covering AI. 
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Don’t Let Regulators 
Ruin Al 

Tech policy scholar Andrea O’Sullivan says the U.S. 
needs to be careful not to hamstring innovation. 


The U.S. has so far been relatively per¬ 
missive toward Al technologies—and we 
should keep it that way. It’s the reason so 
much innovation happens here rather than 
in the more prohibitory European nations. 

The main reason the government 
hasn’t hampered the industry with reg¬ 
ulation is that there’s no overbearing 
federal agency dedicated strictly to Al. 
Instead, we have a patchwork of federal 
and state authorities scrutinizing these 
technologies. The Federal Trade Com¬ 
mission and the National Highway Traf¬ 
fic Safety Administration, for example, 
recently hosted a workshop to determine 
how to oversee automated-car technolo¬ 
gies. The Department of Homeland Secu¬ 
rity has put out reports on potential Al 
threats to critical infrastructure. 

The patchwork approach is imperfect, 
but it has one big benefit—it constrains 
the temptation to regulate excessively. 
Regulators can only apply policies that 
relate to their specialized knowledge. 

But now a growing chorus of academ¬ 
ics and commentators want to kill that 
approach. They’re calling for a whole new 
regulatory body to control Al technolo¬ 
gies. Law professor Frank Pasquale of 
the University of Maryland has called for 
a “Federal Search Commission” similar 
to the FCC to oversee Internet queries. 
Attorney Matthew Scherer, in Portland, 
Oregon, advocates a specialized federal 
Al agency. Law professor Ryan Calo of 
the University of Washington imagines a 
“Federal Robotics Commission.” 


Such ideas are based on the “pre¬ 
cautionary principle”—the idea that an 
innovation must be decelerated or halted 
altogether if a regulator determines that 
the associated risks are too much for 
society to bear. 

Of course, as regulatory scholars 
have long pointed out, the risk analyses 
that regulators employ can be inade¬ 
quate. Imagined or exaggerated risks are 



weighted far more heavily than real bene¬ 
fits, and society is robbed of life-enriching 
(and in many cases life-saving) develop¬ 
ments. Regulators often can’t resist the 
urge to extend their own authority or bud¬ 
gets, regardless of the benefits or costs 
to society. If you give authority to regulate, 
they will regulate. And once you create a 
federal agency, it’s incredibly difficult to 
make it go away. 


As Al grows to touch more and more 
domains of existence, a new federal Al 
agency could have a worryingly large com¬ 
mand over American life. Policymakers 
would need the patience and humility to 
discern one Al application from another. 
The social risks from Al assistants, for 
example, are different from those posed 
by predictive policing software and “smart 
weapons.” But an overly zealous regulatory 
regime might erroneously lump such appli¬ 
cations together, stifling beneficial technol¬ 
ogies while dedicating fewer resources to 
the big problems that really matter. 

The threat to our future and well¬ 
being that precautionary regulation poses, 
meanwhile, is considerable. Al technolo¬ 
gies are poised to generate life-saving 
developments in health and transporta¬ 
tion while modernizing manufacturing and 
trade. The projected economic benefits 
reach the trillions. And on a personal level, 
Al promises to make our lives more com¬ 
fortable and simpler. 

Policymakers who wish to champion 
growth should embrace a stance of “per¬ 
missionless innovation.” Humility, collab¬ 
oration, and voluntary solutions should 
trump the outdated “command and con¬ 
trol” model of the last century. The age 
of smart machines needs a new age of 
smart policy. 


Andrea O’Sullivan is a program manager 
with the Mercatus Center, a free-market- 
oriented think tank at George Mason Uni¬ 
versity’s Technology Policy Program. 
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How We Feel About 
Robots That Feel 

As robots become smart enough to detect our feelings and respond 
appropriately, they could have something like emotions of their own. 
But that won’t necessarily make them more like humans. 

By Louisa Hall 


Octavia, a humanoid robot designed to 

fight fires on Navy ships, has mastered 
an impressive range of facial expressions. 

When she’s turned off, she looks like a 
human-size doll. She has a smooth white 
face with a snub nose. Her plastic eye¬ 
brows sit evenly on her forehead like two 
little capsized canoes. 

When she’s on, however, her eyelids 
fly open and she begins to display emo¬ 
tion. She can nod her head in a gesture of 
understanding; she can widen her eyes 
and lift both her eyebrows in a convincing 
semblance of alarm; or she can cock her 
head to one side and screw up her mouth, 
replicating human confusion. To comic 
effect, she can even arch one eyebrow and 
narrow the opposite eye while tapping her 
metal fingers together, as though plotting 
acts of robotic revenge. 

But Octavia’s range of facial expres¬ 
sions isn’t her most impressive trait. 
What’s amazing is that her emotional 
affect is an accurate response to her inter¬ 
actions with humans. She looks pleased, 
for instance, when she recognizes one of 
her teammates. She looks surprised when a 
teammate gives her a command she wasn’t 
expecting. She looks confused if someone 
says something she doesn’t understand. 

She can show appropriate emotional 
affect because she processes massive 


amounts of information about her envi¬ 
ronment. She can see, hear, and touch. 
She takes visual stock of her surround¬ 
ings using the two cameras built into 
her eyes and analyzes characteristics like 
facial features, complexion, and cloth¬ 
ing. She can detect people’s voices, using 
four microphones and a voice-recognition 
program called Sphinx. She can identify 
25 different objects by 
touch, having learned 
them by using her fingers 
to physically manipulate 
them into various possi¬ 
ble positions and shapes. 

Taken together, these per¬ 
ceptual skills form a part 
of her “embodied cogni¬ 
tive architecture,” which 
allows her—according 
to her creators at the 
Navy Center for Applied 
Research in Artificial 
Intelligence—to “think and act in ways 
similar to people.” 

That’s an exciting claim, but it’s not 
necessarily shocking. We’re accustomed 
to the idea of machines acting like peo¬ 
ple. Automatons created in 18th-century 
France could dance, keep time, and play 
the drums, the dulcimer, or the piano. As 
a kid growing up in the 1980s, I for some 


reason coveted a doll advertised for the 
ability to pee in her pants. 

We’re even accustomed to the idea of 
machines thinking in ways that remind us 
of humans. Many of our long-cherished 
high-water marks for human cognition— 
the ability to beat a grandmaster at chess, 
for example, or to compose a metrically 
accurate sonnet—have been met and sur¬ 
passed by computers. 

Octavia’s actions, how¬ 
ever—the fearful widening 
of her eyes, the confused 
furrow of her plastic eye¬ 
brows—seem to go a step 
further. They imply that 
in addition to thinking the 
way we think, she’s also 
feeling human emotions. 

That’s not really the 
case: Octavia’s emo¬ 
tional affect, according 
to Gregory Trafton, who 
leads the Intelligent Systems Section at 
the Navy AI center, is merely meant to 
demonstrate the kind of thinking she’s 
doing and make it easier for people to 
interact with her. But it’s not always pos¬ 
sible to draw a line between thinking and 
feeling. As Trafton acknowledges, “It’s 
clear that people’s thoughts and emo¬ 
tions are different but impact each other.” 


Culture and Human-Robot 
Interaction in Militarized Spaces: 
A War Story 

By Julie Carpenter 
Routledge,2016 

How Emotions Are Made: 

The Secret Life of the Brain 

By Lisa Feldman Barrett 
Houghton Mifflin Harcourt, 2017 

The U.S. Army Robotic and 
Autonomous Systems Strategy 

March 2017 
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And if, as he says, “emotions influence 
cognition and cognition influences emo¬ 
tion,” Octavia’s ability to think, reason, 
and perceive points to some of the bigger 
questions that will accompany the rise of 
intelligent machines. At what point will 
machines be smart enough to feel some¬ 
thing ? And how would we really know? 

Personalized feelings 

Octavia is programmed with theory of 
mind, meaning that she can anticipate 
the mental states of her human team¬ 
mates. She understands that people have 
potentially conflicting beliefs or intentions. 
When Octavia is given a command that dif¬ 
fers from her expectations, she runs simu¬ 
lations to ascertain what the teammate 
who gave the command might be think¬ 
ing, and why that person thinks this unex¬ 
pected goal is valid. She does this by going 
through her own models of the world but 
altering them slightly, hoping to find one 
that leads to the stated goal. When she tilts 
her head to one side and furrows her eye¬ 
brows, it’s to signal that she’s running these 
simulations, trying to better understand 
her teammate’s beliefs. 

Octavia isn’t programmed with emo¬ 
tional models. Her theory of mind is a 
cognitive pattern. But it functions a lot 
like empathy, that most cherished of all 
human emotions. 

Other robot makers skirt the issue of 
their machines’ emotional intelligence. 
SoftBank Robotics, for instance, which 
sells Pepper—a “pleasant and likeable” 
humanoid robot built to serve as a human 
companion—claims that Pepper can “per¬ 
ceive human emotion,” adding that “Pepper 
loves to interact with you, Pepper wants to 
learn more about your tastes, your habits, 
and quite simply who you are.” But though 
Pepper might have the ability to recognize 
human emotions, and though Pepper might 
be capable of responding with happy smiles 
or expressions of sadness, no one’s claiming 
that Pepper actually feels such emotions. 


What would it take for a robot creator 
to claim that? For one thing, we still don’t 
know everything that’s involved in feeling 
emotions. 

In recent years, revolutions in psychol¬ 
ogy and neuroscience have radically rede¬ 
fined the very concept of emotion, making 
it even more difficult to pin down and 
describe. According to scientists such as 
the psychologist Lisa Feldman Barrett, a 
professor at Northeastern University, it is 
becoming increasingly clear that our emo¬ 
tions vary widely depending on the cul¬ 
ture we’re raised in. They even vary widely 
within an individual in different situa¬ 
tions. In fact, though we share the gen¬ 
eral feelings that make up what’s known 
as “affect” (pleasure, displeasure, arousal, 
and calmness) with most other humans 
and many other animals, our more acute 
and specific emotions vary more than they 
follow particular norms. Fear, for instance, 
is a culturally agreed-upon concept, but it 
plays out in our bodies in myriad ways. It’s 
inspired by different stimuli, it manifests 
differently in our brains, and it’s expressed 
in different ways on our faces. There’s no 
single “fear center” or “fear circuit” in the 
brain, just as there’s no reliably fearful 
facial expression: we all process and dis¬ 
play our fear in radically different ways, 
depending on the situation—ways that, 
through interactions with other people, 
we learn to identify or label as “fear.” 

When we talk about fear, then, we’re 
talking about a generalized umbrella con¬ 
cept rather than something that comes 
out of a specific part of the brain. As 
Barrett puts it, we construct emotions on 
the spot through an interplay of bodily 
systems. So how can we expect program¬ 
mers to accurately model human emotion 
in robots? 

Good soldiers 

There are moral quandaries, as well, 
around programming robots to have 
emotions. These issues are especially 


well demonstrated by military robots 
that, like Octavia, are designed to be sent 
into frightening, painful, or potentially 
lethal situations in place of less dispens¬ 
able human teammates. 

At a 2017 conference sponsored by the 
Army’s somewhat alarmingly titled Mad 
Scientist Initiative, Lieutenant General 
Kevin Mangum, the deputy commander 
for Army Training and Doctrine Com¬ 
mand, specified that such robots should 
and will be autonomous. “As we look at 
our increasingly complex world, there’s 
no doubt that robotics, autonomous sys¬ 
tems, and artificial intelligence will play 
a role,” said Mangum. The 2017 Army 
Robotic and Autonomous Systems Strat¬ 
egy predicts full integration of autono¬ 
mous systems by 2040, replacing today’s 
bomb-disposal robots and other machines 
that are remotely operated by humans. 

When these robots can act and think 
on their own, should they, like Octavia, 
be programmed with the semblance of 
human emotion? Should they be pro¬ 
grammed to actually have human emo¬ 
tion? If we’re sending them into battle, 
should they not only think but feel along¬ 
side their human companions? 

On the one hand, of course not: if 
we’re designing robots with the express 
aim of exposing them to danger, it would 
be sadistic to give them the capacity to 
suffer terror, trauma, or pain. 

On the other hand, however, if emo¬ 
tion affects intelligence and vice versa, 
could we be sure that a robot with no emo¬ 
tion would make a good soldier? What if 
the lack of emotion leads to stupid deci¬ 
sions, unnecessary risks, or excessively 
cruel retribution? Might a robot with no 
emotion decide that the intelligent deci¬ 
sion would be to commit what a human 
soldier would feel was a war crime? Or 
would a robot with no access to fear or 
anger make better decisions than a human 
being would in the same frightening and 
maddening situation? 
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And then there’s the possibility that if 
emotion and intelligence are inextricably 
linked, there’s no such thing as an intel¬ 
ligent robot with no emotion, in which 
case the question of how much emotion 
an autonomous robot should have is, in 
some ways, out of the control of the pro¬ 
grammer dealing with intelligence. 

Emotional attachments 

There’s also the question of how these robots 
might affect their human teammates. 

By 2010, the U.S. Army had begun 
deploying a fleet of about 3,000 small 
tactical robots, largely in response to the 
increasing use of improvised explosive 
devices in warfare. In place of human sol¬ 
diers, these robots trundle down exposed 
roads, into dark caves, and through nar¬ 
row doorways to detect and disable unpre¬ 
dictable IEDs. 

This fleet is composed mostly of 
iRobot’s PackBot and QinetiQ North 
America’s Talon, robots that aren’t par¬ 
ticularly advanced. They look a little like 
WALL-E, their boxy metal bodies bal¬ 
anced on rubber treads that allow them to 
do a pretty good job of crossing rocky ter¬ 
rain, climbing stairs, and making their way 
down dusky hallways. They have jointed 
arms fitted with video cameras to survey 
their surroundings, and claws to tinker 
with explosive devices. 

They’re useful tools, but they’re not 
exactly autonomous. They’re operated 
remotely, like toy cars, by soldiers holding 
devices that are sometimes fitted with joy¬ 
sticks. As an example of AI, the PackBot 
isn’t all that much more advanced than iRo¬ 
bot’s better-known product, the Roomba 
that vacuums under your armchair. 

And yet even now, despite the inex¬ 
pressive nature of these robots, human 
soldiers develop bonds with them. Julie 
Carpenter demonstrates in Culture and 
Human-Robot Interaction in Militarized 
Spaces that these relationships are com¬ 
plicated, both rewarding and painful. 




Even though they can’t convey expressions as Octavia (top left) can, the PackBot (top right) and 
the Talon (below) can bring out loyalty and other kinds of emotional attachment in soldiers who 
work with them. 
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When Carpenter asked one service¬ 
man to describe his feelings about a robot 
that had been destroyed, he responded: 

I mean, it wasn’t obviously ... anywhere 
close to being on the same level as, 
like, you know, a buddy of yours getting 
wounded or seeing a member getting 
taken out or something like that. But 
there was still a certain loss, a sense of 
loss from something happening to one 
of your robots. 

Another serviceman compared his 
robot to a pet dog: 

I mean, you took care of that thing as 
well as you did your team members. And 
you made sure it was cleaned up, and 
made sure the batteries were always 
charged. And if you were not using it, it 
was tucked safely away as best could 
be because you knew if something hap¬ 
pened to the robot, well then, it was 
your turn, and nobody likes to think that. 

Yet another man explained why his 
teammate gave their robot a human name: 

Towards the end of our tour we were 
spending more time outside the wire 
sleeping in our trucks than we were 
inside. We’d sleep inside our trucks out¬ 
side the wire for a good five to six days 
out of the week, and it was three men 
in the truck, you know, one laid across 
the front seats; the other lays across 
the turret. And we can’t download sen¬ 
sitive items and leave them outside the 
truck. Everything has to be locked up, 
so our TALON was in the center aisle 
of our truck and our junior guy named 
it Danielle so he’d have a woman to cud¬ 
dle with at night. 

These men all stress that the robots 
are tools, not living creatures with feel¬ 
ings. Still, they give their robots human 


names and tuck them in safely at night. 
They joke about that impulse, but there’s 
a slightly disturbing dissonance in the 
jokes. The servicemen Carpenter inter¬ 
viewed seem to feel somewhat stuck 
between two feelings: they understand 
the absurdity of caring for an emotion¬ 
less robot that is designed to be expend¬ 
able, but they nevertheless experience the 
temptation to care, at least a little bit. 

Once Carpenter had published her ini¬ 
tial interviews, she received more com¬ 
munication from men and women in the 
military who had developed real bonds 
with their robots. One former explosive 
ordnance disposal technician wrote: 

As I am an EOD technician of eight 
years and three deployments, I can tell 
you that I found your research extremely 
interesting. I can completely agree with 
the other techs you interviewed in say¬ 
ing that the robots are tools and as 
such I will send them into any situation 
regardless of the possible danger. 

However, during a mission in Iraq 
in 2006,1 lost a robot that I had named 
“Stacy 4” (after my wife who is an EOD 
tech as well). She was an excellent robot 
that never gave me any issues, always 
performing flawlessly. Stacy 4 was com¬ 
pletely destroyed and I was only able to 
recover very small pieces of the chas¬ 
sis. Immediately following the blast that 
destroyed Stacy 4,1 can still remember 
the feeling of anger, and lots of it. “My 
beautiful robot was killed ...” was actu¬ 
ally the statement I made to my team 
leader. After the mission was complete 
and I had recovered as much of the 
robot as I could, I cried at the loss of her. 
I felt as if I had lost a dear family mem¬ 
ber. I called my wife that night and told 
her about it too. I know it sounds dumb 
but I still hate thinking about it. I know 
that the robots we use are just machines 
and I would make the same decisions 
again, even knowing the outcome. 


I value human life. I value the rela¬ 
tionships I have with real people. But I 
can tell you that I sure do miss Stacy 4, 
she was a good robot. 

If these are the kinds of testimoni¬ 
als that can be gathered from soldiers 
interacting with faceless machines like 
PackBots and Talons, what would you 
hear from soldiers deployed with robots 
like Octavia, who see and hear and touch 
and can anticipate her human teammates’ 
states of mind? 

In popular conversations about the 
ethics of giving feelings to robots, we tend 
to focus on the effects of such technologi¬ 
cal innovation on the robots themselves. 
Movies and TV shows from Blade Runner 
to Westworld attend to the trauma that 
would be inflicted on feeling robots by 
humans using them for their entertain¬ 
ment. But there is also the inverse to con¬ 
sider: the trauma inflicted on the humans 
who bond with robots and then send them 
to certain deaths. 

What complicates all this even further 
is that if a robot like Octavia ends up feeling 
human emotions, those feelings won’t only 
be the result of the cognitive architecture 
she’s given to start with. If they’re anything 
like our emotions, they’ll evolve in the con¬ 
text of her relationships with her team¬ 
mates, her place in the world she inhabits. 

If her unique robot life, for instance, is 
spent getting sent into fires by her human 
companions, or trundling off alone down 
desert roads laced with explosive devices, 
her emotions will be different from those 
experienced by a more sheltered robot, or 
a more sheltered human. Regardless of 
the recognizable emotional expressions 
she makes, if she spends her life in inhu¬ 
mane situations, her emotions might not 
be recognizably human. 


Louisa Hall, a writer in New York, is 
the author of Speak, a 2015 novel about 
artificial intelligence. 
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The Seven Deadly Sins 
of Al Predictions 

Mistaken extrapolations, limited imagination, and other common mistakes 
that distract us from thinking more productively about the future. 

By Rodney Brooks 


We are surrounded by hysteria about the 

future of artificial intelligence and robot¬ 
ics-hysteria about how powerful they 
will become, how quickly, and what they 
will do to jobs. 

I recently saw a story in MarketWatch 
that said robots will take half of today’s 
jobs in 10 to 20 years. It even had a 
graphic to prove the numbers. 

The claims are ludicrous. (I try to 
maintain professional language, but some¬ 
times ...) For instance, the story appears 
to say that we will go from one million 
grounds and maintenance workers in 
the U.S. to only 50,000 in 10 to 20 years, 
because robots will take over those jobs. 
How many robots are currently opera¬ 
tional in those jobs? Zero. How many real¬ 
istic demonstrations have there been of 
robots working in this arena? Zero. Simi¬ 
lar stories apply to all the other catego¬ 
ries where it is suggested that we will see 
the end of more than 90 percent of jobs 
that currently require physical presence 
at some particular site. 

Mistaken predictions lead to fears 
of things that are not going to happen, 
whether it’s the wide-scale destruction 
of jobs, the Singularity, or the advent of 
AI that has values different from ours 
and might try to destroy us. We need to 
push back on these mistakes. But why are 
people making them? I see seven com¬ 
mon reasons. 

1. Overestimating and underestimating 

Roy Amara was a cofounder of the Insti¬ 
tute for the Future, in Palo Alto, the intel¬ 


lectual heart of Silicon Valley. He is best 
known for his adage now referred to as 
Amara’s Law: 

We tend to overestimate the effect of a 
technology in the short run and under¬ 
estimate the effect in the long run. 

There is a lot wrapped up in these 21 
words. An optimist can read it one way, 
and a pessimist can read it another. 

A great example of the two sides of 
Amara’s Law is the U.S. Global Position¬ 
ing System. Starting in 1978, a constel¬ 
lation of 24 satellites (now 31 including 
spares) were placed in orbit. The goal of 



GPS was to allow precise delivery of muni¬ 
tions by the U.S. military. But the program 
was nearly canceled again and again in 
the 1980s. The first operational use for its 
intended purpose was in 1991 during Des¬ 
ert Storm; it took several more successes 
for the military to accept its utility. 

Today GPS is in what Amara would 
call the long term, and the ways it is used 
were unimagined at first. My Series 2 
Apple Watch uses GPS while I am out 
running, recording my location accu¬ 
rately enough to see which side of the 


street I run along. The tiny size and price 
of the receiver would have been incom¬ 
prehensible to the early GPS engineers. 
The technology synchronizes physics 
experiments across the globe and plays 
an intimate role in synchronizing the U.S. 
electrical grid and keeping it running. It 
even allows the high-frequency traders 
who really control the stock market to 
mostly avoid disastrous timing errors. 
It is used by all our airplanes, large and 
small, to navigate, and it is used to track 
people out of prison on parole. It deter¬ 
mines which seed variant will be planted 
in which part of many fields across the 
globe. It tracks fleets of trucks and reports 
on driver performance. 

GPS started out with one goal, but it 
was a hard slog to get it working as well as 
was originally expected. Now it has seeped 
into so many aspects of our lives that we 
would not just be lost if it went away; we 
would be cold, hungry, and quite possi¬ 
bly dead. 

We see a similar pattern with other 
technologies over the last 30 years. A big 
promise up front, disappointment, and 
then slowly growing confidence in results 
that exceed the original expectations. This 
is true of computation, genome sequenc¬ 
ing, solar power, wind power, and even 
home delivery of groceries. 

AI has been overestimated again and 
again, in the 1960s, in the 1980s, and I 
believe again now, but its prospects for the 
long term are also probably being under¬ 
estimated. The question is: How long is 
the long term ? The next six errors help 
explain why the time scale is being grossly 
underestimated for the future of AI. 

2. Imagining magic 

When I was a teenager, Arthur C. Clarke 
was one of the “big three” science fiction 
writers, along with Robert Heinlein and 
Isaac Asimov. But Clarke was also an 
inventor, a science writer, and a futurist. 
Between 1962 and 1973 he formulated 
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three adages that have come to be known 
as Clarke’s Three Laws: 

When a distinguished but elderly scien¬ 
tist states that something is possible, 
he is almost certainly right. When he 
states that something is impossible, he 
is very probably wrong. 

The only way of discovering the limits 
of the possible is to venture a little way 
past them into the impossible. 

Any sufficiently advanced technology is 
indistinguishable from magic. 

Personally, I should probably be wary 
of the second sentence in his first law, as 
I am much more conservative than some 
others about how quickly AI will be ascen¬ 
dant. But for now I want to expound on 
Clarke’s Third Law. 

Imagine we had a time machine and 
we could transport Isaac Newton from 
the late 17 th century to today, setting him 
down in a place that would be familiar to 
him: Trinity College Chapel at the Univer¬ 
sity of Cambridge. 

Now show Newton an Apple. Pull out 
an iPhone from your pocket, and turn it 
on so that the screen is glowing and full 
of icons, and hand it to him. Newton, who 
revealed how white light is made from 
components of different-colored light by 
pulling apart sunlight with a prism and 
then putting it back together, would no 
doubt be surprised at such a small object 
producing such vivid colors in the dark¬ 
ness of the chapel. Now play a movie of 
an English country scene, and then some 
church music that he would have heard. 
And then show him a Web page with the 
500-plus pages of his personally anno¬ 
tated copy of his masterpiece Principia, 
teaching him how to use the pinch gesture 
to zoom in on details. 

Could Newton begin to explain how 
this small device did all that? Although 
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he invented calculus and explained both 
optics and gravity, he was never able to sort 
out chemistry from alchemy. So I think he 
would be flummoxed, and unable to come 
up with even the barest coherent outline 
of what this device was. It would be no dif¬ 
ferent to him from an embodiment of the 
occult—something that was of great inter¬ 
est to him. It would be indistinguishable 
from magic. And remember, Newton was 
a really smart dude. 

If something is magic, it is hard to know 
its limitations. Suppose we further show 
Newton how the device can illuminate the 
dark, how it can take photos and movies 
and record sound, how it can be used as 
a magnifying glass and as a mirror. Then 
we show him how it can be used to carry 
out arithmetical computations at incred¬ 
ible speed and to many decimal places. We 
show it counting the steps he has taken as 
he carries it, and show him that he can use 
it to talk to people anywhere in the world, 
immediately, from right there in the chapel. 

What else might Newton conjecture 
that the device could do? Prisms work 
forever. Would he conjecture that the 
iPhone would work forever just as it is, 
neglecting to understand that it needs to 
be recharged? Recall that we nabbed him 
from a time 100 years before the birth of 



Michael Faraday, so he lacked a scientific 
understanding of electricity. If the iPhone 
can be a source of light without fire, could 
it perhaps also transmute lead into gold? 

This is a problem we all have with 
imagined future technology. If it is far 
enough away from the technology we have 
and understand today, then we do not 
know its limitations. And if it becomes 


indistinguishable from magic, anything 
one says about it is no longer falsifiable. 

This is a problem I regularly encounter 
when trying to debate with people about 
whether we should fear artificial general 
intelligence, or AGI—the idea that we will 
build autonomous agents that operate 
much like beings in the world. I am told 
that I do not understand how powerful 
AGI will be. That is not an argument. We 
have no idea whether it can even exist. 
I would like it to exist—this has always 
been my own motivation for working in 
robotics and AI. But modern-day AGI 
research is not doing well at all on either 
being general or supporting an indepen¬ 
dent entity with an ongoing existence. It 
mostly seems stuck on the same issues in 
reasoning and common sense that AI has 
had problems with for at least 50 years. 
All the evidence that I see says we have no 
real idea yet how to build one. Its proper¬ 
ties are completely unknown, so rhetori¬ 
cally it quickly becomes magical, powerful 
without limit. 

Nothing in the universe is without 
limit. 

Watch out for arguments about 
future technology that is magical. Such 
an argument can never be refuted. It is 
a faith-based argument, not a scientific 
argument. 

3. Performance versus competence 

We all use cues about how people perform 
some particular task to estimate how well 
they might perform some different task. 
In a foreign city we ask a stranger on the 
street for directions, and she replies with 
confidence and with directions that seem 
to make sense, so we figure we can also 
ask her about the local system for paying 
when you want to take a bus. 

Now suppose a person tells us that 
a particular photo shows people playing 
Frisbee in the park. We naturally assume 
that this person can answer questions like 
What is the shape of a Frisbee? Roughly 
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howfar can a person throw a Frishee? Can 
a person eat a Frishee? Roughly how many 
people play Frishee at once? Can a three- 
month-old person play Frishee?Is today’s 
weather suitable for playing Frishee? 

Computers that can label images like 
“people playing Frisbee in a park” have 
no chance of answering those questions. 
Besides the fact that they can only label 
more images and cannot answer ques¬ 




tions at all, they have no idea what a per¬ 
son is, that parks are usually outside, that 
people have ages, that weather is any¬ 
thing more than how it makes a photo 
look, etc. 

This does not mean that these sys¬ 
tems are useless; they are of great value 
to search engines. But here is what goes 
wrong. People hear that some robot or 
some AI system has performed some task. 
They then generalize from that perfor¬ 
mance to a competence that a person per¬ 
forming the same task could be expected 
to have. And they apply that generaliza¬ 
tion to the robot or AI system. 

Today’s robots and AI systems are 
incredibly narrow in what they can do. 
Human-style generalizations do not apply. 

4. Suitcase words 

Marvin Minsky called words that carry 
a variety of meanings “suitcase words.” 
“Learning” is a powerful suitcase word; 
it can refer to so many different types of 
experience. Learning to use chopsticks is 
a very different experience from learning 
the tune of a new song. And learning to 
write code is a very different experience 
from learning your way around a city. 


When people hear that machine learn¬ 
ing is making great strides in some new 
domain, they tend to use as a mental 
model the way in which a person would 
learn that new domain. However, machine 
learning is very brittle, and it requires lots 
of preparation by human researchers or 
engineers, special-purpose coding, spe¬ 
cial-purpose sets of training data, and a 
custom learning structure for each new 
problem domain. Today’s machine learn¬ 
ing is not at all the sponge-like learning 
that humans engage in, making rapid 
progress in a new domain without having 
to be surgically altered or purpose-built. 

Likewise, when people hear that 
a computer can beat the world chess 
champion (in 1997) or one of the world’s 
best Go players (in 2016), they tend to 
think that it is “playing” the game just 
as a human would. Of course, in real¬ 
ity those programs had no idea what a 
game actually was, or even that they were 
playing. They were also much less adapt¬ 
able. When humans play a game, a small 
change in rules does not throw them off. 
Not so for AlphaGo or Deep Blue. 

Suitcase words mislead people about 
how well machines are doing at tasks that 
people can do. That is partly because AI 
researchers—and, worse, their institu¬ 
tional press offices—are eager to claim 
progress in an instance of a suitcase 



concept. The important phrase here is 
“an instance.” That detail soon gets lost. 
Headlines trumpet the suitcase word, 
and warp the general understanding of 
where AI is and how close it is to accom¬ 
plishing more. 


5. Exponentials 

Many people are suffering from a severe 
case of “exponentialism.” 

Everyone has some idea about Moore’s 
Law, which suggests that computers get 
better and better on a clockwork-like 
schedule. What Gordon Moore actually 
said was that the number of components 
that could fit on a microchip would double 
every year. That held true for 50 years, 
although the time constant for doubling 
gradually lengthened from one year to 
over two years, and the pattern is coming 
to an end. 

Doubling the components on a chip 
has made computers continually dou¬ 
ble in speed. And it has led to memory 

w 

chips that quadruple in capacity every 
two years. It has also led to digital cam¬ 
eras that have better and better resolu¬ 
tion, and LCD screens with exponentially 
more pixels. 

The reason Moore’s Law worked is 
that it applied to a digital abstraction of 
a true-or-false question. In any given cir¬ 
cuit, is there an electrical charge or volt¬ 
age there or not? The answer remains 
clear as chip components get smaller and 
smaller—until a physical limit intervenes, 
and we get down to components with so 
few electrons that quantum effects start to 
dominate. That is where we are now with 
our silicon-based chip technology. 

When people are suffering from expo¬ 
nentialism, they may think that the expo¬ 
nentials they use to justify an argument 
are going to continue apace. But Moore’s 
Law and other seemingly exponential laws 
can fail because they were not truly expo¬ 
nential in the first place. 



83 




















MIT TECHNOLOGY REVIEW 

VOL 120 | NO. 6 


TECHNOLOGYREVIEW.COM 


REVIEWS 


Back in the first part of this century, 
when I was running MIT’s Computer Sci¬ 
ence and Artificial Intelligence Laboratory 
(CSAIL) and needed to help raise money 
for over 90 different research groups, I 
tried to use the memory increase on iPods 
to show sponsors how things were con¬ 
tinuing to change very rapidly. Here are 
the data on how much music storage one 
got in an iPod for $400 or less: 


year 

gigabytes 

2002 

10 

2003 

20 

2004 

40 

2006 

80 

2007 

160 


Then I would extrapolate a few years 
out and ask what we would do with all that 
memory in our pockets. 

Extrapolating through to today, 
we would expect a $400 iPod to have 
160,000 gigabytes of memory. But the 
top iPhone of today (which costs much 
more than $400) has only 256 gigabytes 
of memory, less than double the capacity 
of the 2007 iPod. This particular expo¬ 
nential collapsed very suddenly once the 
amount of memory got to the point where 
it was big enough to hold any reasonable 
person’s music library and apps, photos, 
and videos. Exponentials can collapse 
when a physical limit is hit, or when there 
is no more economic rationale to con¬ 
tinue them. 

Similarly, we have seen a sudden 
increase in performance of AI systems 
thanks to the success of deep learning. Many 
people seem to think that means we will 
continue to see AI performance increase by 
equal multiples on a regular basis. But the 
deep-learning success was 30 years in the 
making, and it was an isolated event. 

That does not mean there will not be 
more isolated events, where work from the 
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backwaters of AI research suddenly fuels 
a rapid-step increase in the performance 
of many AI applications. But there is no 
“law” that says how often they will happen. 

6. Hollywood scenarios 

The plot for many Hollywood science fic¬ 
tion movies is that the world is just as it is 
today, except for one new twist. 

In Bicentennial Man, Richard Martin, 
played by Sam Neill, sits down to break¬ 
fast and is waited upon by a walking, talk¬ 
ing humanoid robot, played by Robin 
Williams. Richard picks up a newspa¬ 
per to read over breakfast. A newspaper! 
Printed on paper. Not a tablet computer, 
not a podcast coming from an Amazon 
Echo-like device, not a direct neural con¬ 
nection to the Internet. 

It turns out that many AI researchers 
and AI pundits, especially those pessimists 
who indulge in predictions about AI get¬ 
ting out of control and killing people, are 
similarly imagination-challenged. They 
ignore the fact that if we are able to even¬ 
tually build such smart devices, the world 
will have changed significantly by then. 
We will not suddenly be surprised by the 
existence of such super-intelligences. They 



will evolve technologically over time, and 
our world will come to be populated by 
many other intelligences, and we will have 
lots of experience already. Long before 
there are evil super-intelligences that want 
to get rid of us, there will be somewhat 
less intelligent, less belligerent machines. 
Before that, there will be really grumpy 
machines. Before that, quite annoying 
machines. And before them, arrogant, 
unpleasant machines. We will change our 


world along the way, adjusting both the 
environment for new technologies and 
the new technologies themselves. I am not 
saying there may not be challenges. I am 
saying that they will not be sudden and 
unexpected, as many people think. 

7. Speed of deployment 

New versions of software are deployed 
very frequently in some industries. New 
features for platforms like Facebook are 
deployed almost hourly. For many new 
features, as long as they have passed inte¬ 
gration testing, there is very little eco¬ 
nomic downside if a problem shows up 
in the field and the version needs to be 
pulled back. This is a tempo that Silicon 
Valley and Web software developers have 
gotten used to. It works because the mar¬ 
ginal cost of newly deploying code is very, 
very close to zero. 

Deploying new hardware, on the other 
hand, has significant marginal costs. We 
know that from our own lives. Many 
of the cars we are buying today, which 
are not self-driving, and mostly are not 
software-enabled, will probably still be 
on the road in the year 2040. This puts 
an inherent limit on how soon all our 
cars will be self-driving. If we build a new 
home today, we can expect that it might 
be around for over 100 years. The build¬ 
ing I live in was built in 1904, and it is 
not nearly the oldest in my neighborhood. 

Capital costs keep physical hardware 
around for a long time, even when there 
are high-tech aspects to it, and even when 
it has an existential mission. 

The U.S. Air Force still flies the B-52H 
variant of the B-52 bomber. This version 
was introduced in 1961, making it 56 
years old. The last one was built in 1962, a 
mere 55 years ago. Currently these planes 
are expected to keep flying until at least 
2040, and perhaps longer—there is talk of 
extending their life to 100 years. 

I regularly see decades-old equipment 
in factories around the world. I even see 
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PCs running Windows 3.0—a software 
version released in 1990. The thinking is 
“If it ain’t broke, don’t fix it.” Those PCs 
and their software have been running the 
same application doing the same task reli¬ 
ably for over two decades. 

The principal control mechanism 
in factories, including brand-new ones 
in the U.S., Europe, Japan, Korea, and 
China, is based on programmable logic 
controllers, or PLCs. These were intro¬ 
duced in 1968 to replace electromechani¬ 
cal relays. The “coil” is still the principal 
abstraction unit used today, and PLCs are 
programmed as though they were a net¬ 
work of 24-volt electromechanical relays. 
Still. Some of the direct wires have been 
replaced by Ethernet cables. But they are 
not part of an open network. Instead they 
are individual cables, run point to point, 
physically embodying the control flow— 
the order in which steps get executed— 
in these brand-new ancient automation 


controllers. When you want to change 
information flow, or control flow, in most 
factories around the world, it takes weeks 
of consultants figuring out what is there, 



designing new reconfigurations, and 
then teams of tradespeople to rewire and 
reconfigure hardware. One of the major 
manufacturers of this equipment recently 
told me that they aim for three software 
upgrades every 20 years. 

In principle, it could be done differ¬ 
ently. In practice, it is not. I just looked on 
a jobs list, and even today, this very day, 
Tesla Motors is trying to hire PLC techni¬ 
cians at its factory in Fremont, California. 


They will use electromagnetic relay emu¬ 
lation in the production of the most AI- 
enhanced automobile that exists. 

A lot of AI researchers and pundits 
imagine that the world is already digi¬ 
tal, and that simply introducing new AI 
systems will immediately trickle down 
to operational changes in the field, in the 
supply chain, on the factory floor, in the 
design of products. 

Nothing could be further from the 
truth. Almost all innovations in robot¬ 
ics and AI take far, far, longer to be really 
widely deployed than people in the field 
and outside the field imagine. 


Rodney Brooks is a former director of 
the Computer Science and Artificial 
Intelligence Laboratory at MIT and a 
founder of Rethink Robotics and iRobot. 
This essay is adapted with permission 
from a post that originally appeared at 
rodneybrooks.com. 


Statement of Ownership, Management and Circulation. (Required by U.S.C. 3685.) (1). Publication Title: MIT Technology Review, (2). Publication No. 535-940, (3). Filing Date: 9/13/2017, (4). Issue 
Frequency: Bi-monthly. (5). No. of Issues Published Annually: 6. (6) Annual Subscription Price: $29.95 (7). Complete Mailing Address of Known Office of Publication: One Main St, 13th Floor, Cambridge, MA 
02142 (8). Complete Mailing Address of Headquarters or General Business Office of Publisher: One Main St, 13th Floor, Cambridge, MA 02142 (9). Full Names and Complete Mailing Addresses of Publisher, 
Editor, and Managing Editor. Publisher: Elizabeth Bramson-Boudreau, One Main St, 13th Floor, Cambridge, MA 02142. Editor: David Rotman, One Main St, 13th Floor, Cambridge, MA 02142. Managing 
Editor: Timothy Maher, One Main St, 13th Floor, Cambridge, MA 02142. (10). Owner: Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA 02138. (11). Known Bondholders, 
Mortgagees, and Other Security Holders Owning or Holding 1 Percent or More of Total Amount of Bonds, Mortgages, or other Securities: None. (12). Tax Status: The purpose, function, and nonprofit status 
of this organization and the exempt status for federal income tax purposes: Has Not Changed During Preceding 12 months. (13). Publication Title: MIT Technology Review. (14) Issue Date for Circulation 
Data Below: July/August 2017. (15). Extent and Nature of Circulation: (a) Total No. of Copies (Net Press Run): Average No. Copies Each Issue During Preceding 12 Months: 210,897; No. Copies of Single Issue 
Published Nearest to Filing Date: 218,208. (b) Legitimate Paid and/or Requested Distribution (1) Mailed Outside-County Paid Subscriptions Stated on PS Form 3541: Average No. Copies Each Issue During 
Preceding 12 Months: 164,758, No. Copies of Single Issue Published Nearest to Filing Date: 169,082. (2) In-County Paid/Requested Mail Subscriptions Stated on PS Form 3541: Average No. Copies Each 
Issue During Preceding 12 Months: 0, No. Copies of Single Issue Published Nearest to Filing Date: 0. (3) Sales Through Dealers and Carriers, Street Vendors, Counter Sales, and Other Paid Distribution 
Outside the USPS: Average No. Copies Each Issue During Preceding 12 Months: 11,856, No. Copies of Single Issue Published Nearest to Filing Date: 11,537. (4) Paid Distribution by Other Mail Classes Through 
the USPS: Average No. Copies Each Issue During Preceding 12 Months: 0, No. Copies of Single Issue Published Nearest to Filing Date: 0. (c) Total Paid and/or Requested Distribution: Average No. Copies 
Each Issue During Preceding 12 Months: 176,623, No. Copies of Single Issue Published Nearest to Filing Date: 180,619. (d) Nonrequested Distribution: (1) Outside-County Nonrequested Copies included 
on PS Form 3541: Average No. Copies Each Issue During Preceding 12 Months: 573, No. Copies of Single Issue Published Nearest to Filing Date: 763. (2) In-County Nonrequested Copies included on PS 
Form 3541: Average No. Copies Each Issue During Preceding 12 Months: 0, No. Copies of Single Issue Published Nearest to Filing Date: 0. (3) Nonrequested Copies Distributed Through the USPS by Other 
Classes of Mail: 0 Average No. Copies Each Issue During Preceding 12 Months: 0, No. Copies of Single Issue Published Nearest to Filing Date: 0 (4) Nonrequested Copies Distributed Outside the Mail: 
Average No. Copies Each Issue During Preceding 12 Months: 4,581, No. Copies of Single Issue Published Nearest to Filing Date: 6764. (e) Total Free or Nominal Rate Distribution: Average No. Copies Each 
Issue During Preceding 12 Months: 5,154, No. Copies of Single Issue Published Nearest to Filing Date: 7527. (f) Total Distribution: Average No. Copies Each Issue During Preceding 12 Months: 181,777, No. 
Copies of Single Issue Published Nearest to Filing Date: 188146. (g) Copies not Distributed: Average No. Copies Each Issue During Preceding 12 Months: 29,120, No. Copies of Single Issue Published Nearest 
to Filing Date: 30,062. (h) Total: Average No. Copies Each Issue During Preceding 12 Months: 210,897, No. Copies of Single Issue Published Nearest to Filing Date: 218,208. (i) Percent Paid: Average No. 
Copies Each Issue During Preceding 12 Months: 97.2%, No. Copies of Single Issue Published Nearest to Filing Date: 96.0%. This Statement of Ownership will be printed in the November/December issue 
of this publication. I certify that all the information furnished on this form is true and complete. Electronic Copy Circulation: Requested and Paid Electronic Copies: Average No. Copies Each Issue During 
Preceding 12 Months: 14,989 No. Copies of Single Issue Published Nearest to Filing Date: 15,039 Total requested and paid copies: Average No. Copies Each Issue During Preceding 12 Months: 176,623 No. 
Copies of Single Issue Published Nearest to Filing Date: 180,619. Total Requested Copy Distribution (Line 15F) + Requested Paid Electronic Copies: Average No. Copies Each Issue During Preceding 12 
Months: 181,776 No. Copies of Single Issue Published Nearest to Filing Date: 188,146. Percent Paid and /or Reqested (16B divided by 16c x 100): Average No. Copies Each Issue During Preceding 12 Months: 
97.2% No. Copies of Single Issue Published Nearest to Filing Date: 96% Doreen Adger, SVP of Consumer Revenues and Marketing. 


86 















Finally, one application that syncs 
your PDF articles, your lab group, 
and your next manuscript. 

ACS ChemWorx is a free, single-source solution 
designed to increase productivity in research 
management, collaboration, and publishing. 

Find out how you can spend less time 
organizing and more time researching. 

m — m 

Store. Publish. 


Login today at www.c 


- Q,- 

Search. 


Import. 


Annotate. 




Collaborate. 


Import your reference library 
using your existing file structure. 



ACS 


ChemWorx 

RESEARCH. COLLABORATE. PUBLISH. 


Available onthe^^J GET IT ON 

m AppStore ^ Google play 



ACS Publications 


MOST TRUSTED. MOST CITED. MOST READ. 





















MIT TECHNOLOGY REVIEW 

VOL 120 | NO. 6 


TECHNOLOGYREVIEW.COM 


THE Al ISSUE 



Trevor Paglen 

A Man (Corpus: The Humans) 
from the series 

Adversarially Evolved Hallucination 
2017 


What do machine hallucinations look like? Trevor Paglen, who has been an artist in 
residence at Stanford, unearths them by training facial-recognition systems on particular 
sets of images. This picture is one that a computer recognized as a man after Paglen fed it 
images associated with omens, portents, monsters, and dreams. Sure, giving the machine 
those training images is a bit of a trick, but Paglen’s point is that any data fed to Al is chosen 
subjectively. Computers do not see the world precisely as it is; they see it within the limits of 
the data given to them to represent the world. That’s a flaw Al and people have in common. 
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